[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:09 dist-test-slave-f9rk krb5kdc[22115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:09 dist-test-slave-f9rk krb5kdc[22115](info): set up 2 sockets
Jun 19 20:27:09 dist-test-slave-f9rk krb5kdc[22115](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 20:27:10.280354 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:10 dist-test-slave-f9rk krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:10.394579 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:45823
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:45823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:10.473256 22128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:10.473506 22128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:10.473546 22128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:10.474704 22128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:10.474771 22128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:10.474795 22128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:10.474843 22128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:10.474881 22128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:10.476450 22128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:45823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:45823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:10.476730 22128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:10.476970 22128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:10.481779 22133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:10.481817 22136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:10.481794 22134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:10.482067 22128 server_base.cc:1034] running on GCE node
I20260619 20:27:10.482326 22128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:10.482653 22128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:10.483816 22128 hybrid_clock.cc:648] HybridClock initialized: now 1781900830483814 us; error 53 us; skew 500 ppm
Jun 19 20:27:10 dist-test-slave-f9rk krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900830, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:10.486158 22128 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:10.486706 22128 webserver.cc:460] Webserver started at http://127.21.151.62:35153/ using document root <none> and password file <none>
I20260619 20:27:10.486940 22128 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:10.487015 22128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:10.487187 22128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:10.488338 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "d41062b8602542e8a3cf404b0a26c59c"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "b9f22e83d7847457c96447021a6add1d"
server_key_iv: "75141f6140d3650aa18076b8a1d7408f"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.488713 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "d41062b8602542e8a3cf404b0a26c59c"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "b9f22e83d7847457c96447021a6add1d"
server_key_iv: "75141f6140d3650aa18076b8a1d7408f"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.490082 22128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:10.490783 22143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:10.490949 22128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:10.491056 22128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "d41062b8602542e8a3cf404b0a26c59c"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "b9f22e83d7847457c96447021a6add1d"
server_key_iv: "75141f6140d3650aa18076b8a1d7408f"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.491137 22128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:10.501958 22128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:10.503806 22128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:10.503969 22128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:10.508052 22128 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:45823
I20260619 20:27:10.508086 22195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:45823 every 8 connection(s)
I20260619 20:27:10.508458 22128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:10.508929 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 22128
I20260619 20:27:10.509011 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:10.509618 22196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:10.509639 22108 external_mini_cluster.cc:1428] Setting key 93d804a9fdae5e7de34e6d283040f737
I20260619 20:27:10.513182 22196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c: Bootstrap starting.
I20260619 20:27:10.513942 22196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:10.514308 22196 log.cc:824] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:10.515125 22196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c: No bootstrap required, opened a new log
I20260619 20:27:10.517915 22196 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41062b8602542e8a3cf404b0a26c59c" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45823 } }
I20260619 20:27:10.518106 22196 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:10.518149 22196 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41062b8602542e8a3cf404b0a26c59c, State: Initialized, Role: FOLLOWER
I20260619 20:27:10.518357 22196 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41062b8602542e8a3cf404b0a26c59c" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45823 } }
I20260619 20:27:10.518468 22196 raft_consensus.cc:388] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:10.518507 22196 raft_consensus.cc:482] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:10.518566 22196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:10.519338 22196 raft_consensus.cc:504] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41062b8602542e8a3cf404b0a26c59c" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45823 } }
I20260619 20:27:10.519467 22196 leader_election.cc:302] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41062b8602542e8a3cf404b0a26c59c; no voters:
I20260619 20:27:10.519685 22196 leader_election.cc:288] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:10.519719 22201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:10.519853 22201 raft_consensus.cc:686] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [term 1 LEADER]: Becoming Leader. State: Replica: d41062b8602542e8a3cf404b0a26c59c, State: Running, Role: LEADER
I20260619 20:27:10.519997 22201 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41062b8602542e8a3cf404b0a26c59c" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45823 } }
I20260619 20:27:10.520040 22196 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:10.520422 22202 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41062b8602542e8a3cf404b0a26c59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41062b8602542e8a3cf404b0a26c59c" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45823 } } }
I20260619 20:27:10.520519 22202 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:10.520794 22206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:10.521358 22206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:10.521723 22203 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41062b8602542e8a3cf404b0a26c59c. Latest consensus state: current_term: 1 leader_uuid: "d41062b8602542e8a3cf404b0a26c59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41062b8602542e8a3cf404b0a26c59c" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45823 } } }
I20260619 20:27:10.522019 22203 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:10.523104 22206 catalog_manager.cc:1347] Generated new cluster ID: 80b27e5c233841179b584480e90eb1f0
I20260619 20:27:10.523159 22206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:27:10 dist-test-slave-f9rk krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:10.528759 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:10.510585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.21.151.62:45823)
0619 20:27:10.510800 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:10.510808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:10.510956 (+ 148us) server_negotiation.cc:407] Connection header received
0619 20:27:10.514099 (+ 3143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:10.514105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:10.514151 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:10.514254 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:10.516662 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:10.517584 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:10.519229 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:10.519808 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:10.519927 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:10.524832 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:10.524836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:10.524836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:10.524846 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:10.526787 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:10.527327 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:10.527330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:10.527331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:10.527424 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:10.527711 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:10.527711 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:10.527712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:10.527992 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:10.528138 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:10.528434 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 20:27:10.528557 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":60,"threads_started":1}
I20260619 20:27:10.543152 22206 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:10.543876 22206 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:10.557329 22206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d41062b8602542e8a3cf404b0a26c59c: Generated new TSK 0
I20260619 20:27:10.557515 22206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:27:10.587441 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45823
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:10.666340 22222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:10.666563 22222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:10.666601 22222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:10.667843 22222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:10.667899 22222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:10.667943 22222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:10.669358 22222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:10.669620 22222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:10.669839 22222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:10.674577 22230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:10.674552 22228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:10.674638 22227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:10.674796 22222 server_base.cc:1034] running on GCE node
I20260619 20:27:10.674985 22222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:10.675222 22222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:10.676376 22222 hybrid_clock.cc:648] HybridClock initialized: now 1781900830676363 us; error 43 us; skew 500 ppm
Jun 19 20:27:10 dist-test-slave-f9rk krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900830, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:10.678881 22222 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:27:10.679280 22222 webserver.cc:460] Webserver started at http://127.21.151.1:45653/ using document root <none> and password file <none>
I20260619 20:27:10.679498 22222 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:10.679565 22222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:10.679746 22222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:10.680764 22222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "e537b16d1765436aac2241be193b17d7"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "4370e89f1f8cb1e8100453317007be6f"
server_key_iv: "b8c52d01843fbc535600d28870e3c130"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.681113 22222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "e537b16d1765436aac2241be193b17d7"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "4370e89f1f8cb1e8100453317007be6f"
server_key_iv: "b8c52d01843fbc535600d28870e3c130"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.682454 22222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:10.683090 22237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:10.683252 22222 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:10.683363 22222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "e537b16d1765436aac2241be193b17d7"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "4370e89f1f8cb1e8100453317007be6f"
server_key_iv: "b8c52d01843fbc535600d28870e3c130"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.683447 22222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:10.700723 22222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:10.702968 22222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:10.703136 22222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:10.703379 22222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:10.703744 22222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:10.703796 22222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:10.703837 22222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:10.703866 22222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:10.709321 22222 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:46071
I20260619 20:27:10.709360 22350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:46071 every 8 connection(s)
I20260619 20:27:10.709692 22222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:27:10.712026 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 22222
I20260619 20:27:10.712100 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:10.712260 22108 external_mini_cluster.cc:1428] Setting key 695ac2b535a69bc23a2e791b5a2d9445
Jun 19 20:27:10 dist-test-slave-f9rk krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900830, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:10.720399 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:10.710562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:44797 (local address 127.21.151.62:45823)
0619 20:27:10.710637 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:10.710639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:10.710833 (+ 194us) server_negotiation.cc:407] Connection header received
0619 20:27:10.711292 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:10.711293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:10.711302 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:10.711362 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:10.712531 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:10.713186 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:10.714612 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:10.714829 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:10.714916 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:10.717481 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:10.717482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:10.717482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:10.717488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:10.718830 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:10.719305 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:10.719306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:10.719306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:10.719354 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:10.719706 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:10.719707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:10.719707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:10.719808 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:10.719874 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:10.720269 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 20:27:10.720337 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
I20260619 20:27:10.720649 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:10.710544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45823 (local address 127.21.151.1:44797)
0619 20:27:10.710701 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:10.710749 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:10.711212 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:10.711414 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:10.711417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:10.712193 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:10.712430 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:10.712435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:10.713274 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:10.713275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:10.714521 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:10.714524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:10.714925 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:10.714927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:10.715064 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:10.715572 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:10.715580 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:10.717386 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:10.718913 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:10.718921 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:10.718923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:10.719232 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:10.719419 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:10.719420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:10.719421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:10.719587 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:10.719892 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:10.719899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:10.720109 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 20:27:10.720233 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 20:27:10.720386 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260619 20:27:10.721174 22351 heartbeater.cc:347] Connected to a master server at 127.21.151.62:45823
I20260619 20:27:10.721292 22351 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:10.721462 22351 heartbeater.cc:511] Master 127.21.151.62:45823 requested a full tablet report, sending...
I20260619 20:27:10.722018 22160 ts_manager.cc:194] Registered new tserver with Master: e537b16d1765436aac2241be193b17d7 (127.21.151.1:46071)
I20260619 20:27:10.722685 22160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:44797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:27:10.738238 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45823
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:10.817368 22356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:10.817600 22356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:10.817641 22356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:10.818792 22356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:10.818858 22356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:10.818903 22356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:10.820397 22356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:10.820744 22356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:10.820940 22356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:10.825961 22361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:10.826025 22362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:10.826108 22364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:10.825982 22356 server_base.cc:1034] running on GCE node
I20260619 20:27:10.826326 22356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:10.826573 22356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:10.827726 22356 hybrid_clock.cc:648] HybridClock initialized: now 1781900830827696 us; error 64 us; skew 500 ppm
Jun 19 20:27:10 dist-test-slave-f9rk krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900830, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:10.830214 22356 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:27:10.830674 22356 webserver.cc:460] Webserver started at http://127.21.151.2:42767/ using document root <none> and password file <none>
I20260619 20:27:10.830854 22356 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:10.830893 22356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:10.830981 22356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:10.832062 22356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "41e4b15db00b49829075cdcc32c7a0e9"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "be33d35f15807e733b74744b5be8c0db"
server_key_iv: "a62d3ff29035acf60e171f46dea88c26"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.832429 22356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "41e4b15db00b49829075cdcc32c7a0e9"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "be33d35f15807e733b74744b5be8c0db"
server_key_iv: "a62d3ff29035acf60e171f46dea88c26"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.833642 22356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:10.834244 22371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:10.834399 22356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:10.834479 22356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "41e4b15db00b49829075cdcc32c7a0e9"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "be33d35f15807e733b74744b5be8c0db"
server_key_iv: "a62d3ff29035acf60e171f46dea88c26"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.834550 22356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:10.854467 22356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:10.856316 22356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:10.856478 22356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:10.856703 22356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:10.857021 22356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:10.857076 22356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:10.857110 22356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:10.857139 22356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:10.862766 22356 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:39157
I20260619 20:27:10.862814 22484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:39157 every 8 connection(s)
I20260619 20:27:10.863152 22356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:27:10 dist-test-slave-f9rk krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900830, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:10.872015 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:10.863891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:45271 (local address 127.21.151.62:45823)
0619 20:27:10.863962 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:10.863964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:10.864169 (+ 205us) server_negotiation.cc:407] Connection header received
0619 20:27:10.864693 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:10.864693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:10.864703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:10.864757 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:10.865630 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:10.866115 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:10.867171 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:10.867446 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:10.867515 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:10.869640 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:10.869641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:10.869642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:10.869647 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:10.870846 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:10.871199 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:10.871201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:10.871201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:10.871266 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:10.871466 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:10.871467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:10.871467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:10.871573 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:10.871674 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:10.871890 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 20:27:10.871952 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 20:27:10.872193 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:10.863859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45823 (local address 127.21.151.2:45271)
0619 20:27:10.864025 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:10.864067 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:10.864604 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:10.864811 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:10.864814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:10.865349 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:10.865534 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:10.865538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:10.866191 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:10.866192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:10.866942 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:10.866943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:10.867529 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:10.867529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:10.867642 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:10.868226 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:10.868234 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:10.869563 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:10.870910 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:10.870915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:10.870917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:10.871121 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:10.871323 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:10.871324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:10.871324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:10.871413 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:10.871668 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:10.871673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:10.871781 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 20:27:10.871862 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 20:27:10.871972 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260619 20:27:10.872751 22485 heartbeater.cc:347] Connected to a master server at 127.21.151.62:45823
I20260619 20:27:10.872852 22485 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:10.872877 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 22356
I20260619 20:27:10.872954 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:10.873044 22485 heartbeater.cc:511] Master 127.21.151.62:45823 requested a full tablet report, sending...
I20260619 20:27:10.873142 22108 external_mini_cluster.cc:1428] Setting key 9419f9753faa5459115e5e6171c2eaf1
I20260619 20:27:10.873449 22160 ts_manager.cc:194] Registered new tserver with Master: 41e4b15db00b49829075cdcc32c7a0e9 (127.21.151.2:39157)
I20260619 20:27:10.873983 22160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:45271
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:27:10.896487 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45823
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:10.974809 22490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:10.975008 22490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:10.975031 22490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:10.976092 22490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:10.976125 22490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:10.976150 22490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:10.977434 22490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:10.977597 22490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:10.977757 22490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:27:10.982682 22490 server_base.cc:1034] running on GCE node
W20260619 20:27:10.982642 22496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:10.982656 22498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:10.982642 22495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:10.983242 22490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:10.983520 22490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:10.984678 22490 hybrid_clock.cc:648] HybridClock initialized: now 1781900830984667 us; error 25 us; skew 500 ppm
Jun 19 20:27:10 dist-test-slave-f9rk krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900830, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:10.987111 22490 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:27:10.987526 22490 webserver.cc:460] Webserver started at http://127.21.151.3:45173/ using document root <none> and password file <none>
I20260619 20:27:10.987766 22490 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:10.987810 22490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:10.987905 22490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:10.988916 22490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "f6e2b491c9f6418c9d2f636299d1c50a"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "1cbbaacfead1dfdaf28df47d95801b52"
server_key_iv: "698a75459776198d8110303db1496f82"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.989212 22490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "f6e2b491c9f6418c9d2f636299d1c50a"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "1cbbaacfead1dfdaf28df47d95801b52"
server_key_iv: "698a75459776198d8110303db1496f82"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.990465 22490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:27:10.991175 22505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:10.991365 22490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:10.991475 22490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "f6e2b491c9f6418c9d2f636299d1c50a"
format_stamp: "Formatted at 2026-06-19 20:27:10 on dist-test-slave-f9rk"
server_key: "1cbbaacfead1dfdaf28df47d95801b52"
server_key_iv: "698a75459776198d8110303db1496f82"
server_key_version: "encryptionkey@0"
I20260619 20:27:10.991547 22490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:11.016547 22490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:11.019013 22490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:11.019182 22490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:11.019425 22490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:11.019770 22490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:11.019826 22490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:11.019901 22490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:11.019946 22490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:11.026077 22490 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:45201
I20260619 20:27:11.026132 22618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:45201 every 8 connection(s)
I20260619 20:27:11.026515 22490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:11.031018 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 22490
I20260619 20:27:11.031104 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:11.031260 22108 external_mini_cluster.cc:1428] Setting key 369180e5c0fbf5f0d8a7de57bfaa3178
Jun 19 20:27:11 dist-test-slave-f9rk krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900830, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:11.036121 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:11.027225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:40037 (local address 127.21.151.62:45823)
0619 20:27:11.027302 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:11.027304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:11.027500 (+ 196us) server_negotiation.cc:407] Connection header received
0619 20:27:11.028108 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:11.028109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:11.028122 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:11.028171 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:11.029163 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:11.029672 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:11.030679 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:11.030963 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:11.031038 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:11.033623 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:11.033625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:11.033625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:11.033631 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:11.034895 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:11.035266 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:11.035267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:11.035267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:11.035316 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:11.035531 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:11.035532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:11.035532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:11.035686 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:11.035750 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:11.035989 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 20:27:11.036050 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:27:11.036335 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:11.027194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45823 (local address 127.21.151.3:40037)
0619 20:27:11.027362 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:11.027406 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:11.027976 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:11.028239 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:11.028243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:11.028860 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:11.029063 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:11.029068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:11.029770 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:11.029771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:11.030572 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:11.030574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:11.031189 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:11.031191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:11.031365 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:11.032015 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:11.032026 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:11.033540 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:11.034966 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:11.034972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:11.034974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:11.035198 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:11.035372 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:11.035373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:11.035373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:11.035469 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:11.035752 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:11.035757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:11.035873 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 20:27:11.035962 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 20:27:11.036086 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260619 20:27:11.036779 22619 heartbeater.cc:347] Connected to a master server at 127.21.151.62:45823
I20260619 20:27:11.036861 22619 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:11.037060 22619 heartbeater.cc:511] Master 127.21.151.62:45823 requested a full tablet report, sending...
I20260619 20:27:11.037403 22160 ts_manager.cc:194] Registered new tserver with Master: f6e2b491c9f6418c9d2f636299d1c50a (127.21.151.3:45201)
I20260619 20:27:11.037739 22160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:40037
I20260619 20:27:11.043123 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:27:11 dist-test-slave-f9rk krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:27:11 dist-test-slave-f9rk krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.151.1@KRBTEST.COM
I20260619 20:27:11.066188 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:11.057878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.21.151.1:46071)
0619 20:27:11.058036 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:11.058042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:11.058060 (+ 18us) server_negotiation.cc:407] Connection header received
0619 20:27:11.058087 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:11.058089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:11.058102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:11.058168 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:11.058688 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:11.059457 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:11.060204 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:11.060533 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:11.060588 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:11.062775 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:11.062783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:11.062783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:11.062790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:11.064748 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:11.065186 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:11.065190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:11.065190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:11.065294 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:11.065504 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:11.065505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:11.065505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:11.065637 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:11.065814 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:11.065909 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 20:27:11.066004 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260619 20:27:11.066722 22265 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46230
Jun 19 20:27:11 dist-test-slave-f9rk krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:27:11 dist-test-slave-f9rk krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.1@KRBTEST.COM
I20260619 20:27:11.090145 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:11.083443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.21.151.1:46071)
0619 20:27:11.083492 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:11.083495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:11.083566 (+ 71us) server_negotiation.cc:407] Connection header received
0619 20:27:11.083693 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:11.083694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:11.083706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:11.083769 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:11.084310 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:11.084753 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:11.085495 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:11.085684 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:11.085778 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:11.088101 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:11.088102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:11.088102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:11.088108 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:11.089286 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:11.089584 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:11.089585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:11.089585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:11.089631 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:11.089787 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:11.089788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:11.089788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:11.089869 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:11.089948 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:11.090022 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 20:27:11.090081 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:27:11.090688 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 22222
I20260619 20:27:11.095536 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 22356
I20260619 20:27:11.101540 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 22490
I20260619 20:27:11.105743 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 22128
2026-06-19T20:27:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1866 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:11 dist-test-slave-f9rk krb5kdc[22633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:11 dist-test-slave-f9rk krb5kdc[22633](info): set up 2 sockets
Jun 19 20:27:11 dist-test-slave-f9rk krb5kdc[22633](info): commencing operation
krb5kdc: starting...
W20260619 20:27:12.136202 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:12 dist-test-slave-f9rk krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:12.249979 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:41015
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:41015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:12.328717 22646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:12.328919 22646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:12.328941 22646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:12.330195 22646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:12.330237 22646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:12.330251 22646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:12.330265 22646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:12.330322 22646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:12.331748 22646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:41015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:12.331990 22646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:12.332226 22646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:12.336830 22652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:12.336935 22646 server_base.cc:1034] running on GCE node
W20260619 20:27:12.336853 22654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:12.336905 22651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:12.337260 22646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:12.337602 22646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:12.338780 22646 hybrid_clock.cc:648] HybridClock initialized: now 1781900832338761 us; error 35 us; skew 500 ppm
Jun 19 20:27:12 dist-test-slave-f9rk krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900832, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:12.341806 22646 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:12.342268 22646 webserver.cc:460] Webserver started at http://127.21.151.62:40497/ using document root <none> and password file <none>
I20260619 20:27:12.342491 22646 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:12.342535 22646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:12.342676 22646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:12.343775 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "89e94ac34aca44b5a7ffa4431cc57150"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "d9e266442a56dd3c3c2e2b3464762c33"
server_key_iv: "6523c5e9f3fb17c5a077f4fb47245d9f"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.344131 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "89e94ac34aca44b5a7ffa4431cc57150"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "d9e266442a56dd3c3c2e2b3464762c33"
server_key_iv: "6523c5e9f3fb17c5a077f4fb47245d9f"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.345397 22646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:12.346051 22661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.346202 22646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:12.346293 22646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "89e94ac34aca44b5a7ffa4431cc57150"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "d9e266442a56dd3c3c2e2b3464762c33"
server_key_iv: "6523c5e9f3fb17c5a077f4fb47245d9f"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.346346 22646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:12.356563 22646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:12.358244 22646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:12.358348 22646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:12.362028 22646 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:41015
I20260619 20:27:12.362092 22713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:41015 every 8 connection(s)
I20260619 20:27:12.362411 22646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:12.363279 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:12.364212 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 22646
I20260619 20:27:12.364275 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:12.364434 22108 external_mini_cluster.cc:1428] Setting key f3c84c6e007cf7161604011e4e5c0619
I20260619 20:27:12.366359 22714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150: Bootstrap starting.
I20260619 20:27:12.367025 22714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:12.367343 22714 log.cc:824] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:12.368079 22714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150: No bootstrap required, opened a new log
I20260619 20:27:12.370163 22714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e94ac34aca44b5a7ffa4431cc57150" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41015 } }
I20260619 20:27:12.370291 22714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:12.370312 22714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e94ac34aca44b5a7ffa4431cc57150, State: Initialized, Role: FOLLOWER
I20260619 20:27:12.370445 22714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e94ac34aca44b5a7ffa4431cc57150" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41015 } }
I20260619 20:27:12.370524 22714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:12.370548 22714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:12.370630 22714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:12.371134 22714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e94ac34aca44b5a7ffa4431cc57150" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41015 } }
Jun 19 20:27:12 dist-test-slave-f9rk krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:12.371258 22714 leader_election.cc:302] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e94ac34aca44b5a7ffa4431cc57150; no voters:
I20260619 20:27:12.371479 22714 leader_election.cc:288] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:12.371510 22719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:12.371649 22719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [term 1 LEADER]: Becoming Leader. State: Replica: 89e94ac34aca44b5a7ffa4431cc57150, State: Running, Role: LEADER
I20260619 20:27:12.371770 22714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:12.371793 22719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e94ac34aca44b5a7ffa4431cc57150" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41015 } }
I20260619 20:27:12.372129 22719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89e94ac34aca44b5a7ffa4431cc57150. Latest consensus state: current_term: 1 leader_uuid: "89e94ac34aca44b5a7ffa4431cc57150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e94ac34aca44b5a7ffa4431cc57150" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41015 } } }
I20260619 20:27:12.372203 22719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:12.372404 22721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89e94ac34aca44b5a7ffa4431cc57150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e94ac34aca44b5a7ffa4431cc57150" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41015 } } }
I20260619 20:27:12.372552 22721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:12.372785 22723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:12.374203 22723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:12.375511 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.364940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.21.151.62:41015)
0619 20:27:12.365118 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:12.365128 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:12.365158 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:27:12.365617 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:12.365626 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:12.365686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:12.365825 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:12.366367 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.367785 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.368562 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.369194 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.369307 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.371818 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:12.371823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:12.371823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:12.371834 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:12.373716 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:12.374056 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:12.374059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:12.374059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:12.374160 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:12.374399 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:12.374400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:12.374400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:12.374699 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:12.374844 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:12.375250 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 20:27:12.375367 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260619 20:27:12.376024 22723 catalog_manager.cc:1347] Generated new cluster ID: 318498b5ef4244dab28f576513e62318
I20260619 20:27:12.376097 22723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:12.395813 22723 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:12.396538 22723 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:12.409353 22723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89e94ac34aca44b5a7ffa4431cc57150: Generated new TSK 0
I20260619 20:27:12.409574 22723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:27:12.436245 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41015
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:12.515810 22740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:12.516007 22740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:12.516031 22740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:12.517156 22740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:12.517208 22740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:12.517234 22740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:12.518626 22740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:12.518826 22740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:12.519066 22740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:12.523983 22748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:12.524179 22740 server_base.cc:1034] running on GCE node
W20260619 20:27:12.524152 22745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:12.523998 22746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:12.524535 22740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:12.524798 22740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:12.525964 22740 hybrid_clock.cc:648] HybridClock initialized: now 1781900832525956 us; error 38 us; skew 500 ppm
Jun 19 20:27:12 dist-test-slave-f9rk krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900832, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:12.528458 22740 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:27:12.528878 22740 webserver.cc:460] Webserver started at http://127.21.151.1:42375/ using document root <none> and password file <none>
I20260619 20:27:12.529111 22740 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:12.529181 22740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:12.529331 22740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:12.530431 22740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "453a796931fa4de2ba0a5d02b73ad308"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "df0e76fa56bb5c8dec04bfba650742b8"
server_key_iv: "6450619a1063ec8db156c59d409ba217"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.530815 22740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "453a796931fa4de2ba0a5d02b73ad308"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "df0e76fa56bb5c8dec04bfba650742b8"
server_key_iv: "6450619a1063ec8db156c59d409ba217"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.532198 22740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:27:12.532905 22755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.533078 22740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.533180 22740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "453a796931fa4de2ba0a5d02b73ad308"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "df0e76fa56bb5c8dec04bfba650742b8"
server_key_iv: "6450619a1063ec8db156c59d409ba217"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.533259 22740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:12.545753 22740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:12.547443 22740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:12.547600 22740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:12.547885 22740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:12.548192 22740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:12.548244 22740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.548282 22740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:12.548309 22740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.554297 22740 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:40921
I20260619 20:27:12.554351 22868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:40921 every 8 connection(s)
I20260619 20:27:12.554633 22740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:27:12 dist-test-slave-f9rk krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900832, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:12.560690 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 22740
I20260619 20:27:12.560763 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:12.560933 22108 external_mini_cluster.cc:1428] Setting key f5245cd07c9176a7c62e95904f2d6892
I20260619 20:27:12.563863 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.555250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:59743 (local address 127.21.151.62:41015)
0619 20:27:12.555309 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:12.555311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:12.555522 (+ 211us) server_negotiation.cc:407] Connection header received
0619 20:27:12.556501 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:12.556502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:12.556513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:12.556570 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:12.557170 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.557685 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.558626 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.558817 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.558867 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.561026 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:12.561028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:12.561029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:12.561038 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:12.562468 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:12.562887 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:12.562888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:12.562889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:12.562948 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:12.563199 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:12.563201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:12.563201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:12.563315 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:12.563394 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:12.563681 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 20:27:12.563768 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:27:12.564071 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.555225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:41015 (local address 127.21.151.1:59743)
0619 20:27:12.555399 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:12.555438 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:12.556406 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:12.556642 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:12.556648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:12.556919 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:12.557099 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.557103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.557759 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.557760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.558555 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.558557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.558887 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.558887 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.558980 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.559401 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:12.559408 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:12.560917 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:12.562563 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:12.562570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:12.562572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:12.562795 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:12.563019 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:12.563020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:12.563021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:12.563132 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:12.563384 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:12.563390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:12.563530 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 20:27:12.563674 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 20:27:12.563829 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260619 20:27:12.564508 22869 heartbeater.cc:347] Connected to a master server at 127.21.151.62:41015
I20260619 20:27:12.564585 22869 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:12.564774 22869 heartbeater.cc:511] Master 127.21.151.62:41015 requested a full tablet report, sending...
I20260619 20:27:12.565335 22678 ts_manager.cc:194] Registered new tserver with Master: 453a796931fa4de2ba0a5d02b73ad308 (127.21.151.1:40921)
I20260619 20:27:12.566314 22678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:59743
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:27:12.585678 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41015
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:12.664750 22874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:12.665016 22874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:12.665055 22874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:12.666177 22874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:12.666244 22874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:12.666290 22874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:12.667729 22874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:12.667969 22874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:12.668210 22874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:12.673205 22880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:12.673177 22882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:12.673214 22879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:12.673270 22874 server_base.cc:1034] running on GCE node
I20260619 20:27:12.673600 22874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:12.673851 22874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:12.675020 22874 hybrid_clock.cc:648] HybridClock initialized: now 1781900832675008 us; error 27 us; skew 500 ppm
Jun 19 20:27:12 dist-test-slave-f9rk krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900832, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:12.677372 22874 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:27:12.677800 22874 webserver.cc:460] Webserver started at http://127.21.151.2:38657/ using document root <none> and password file <none>
I20260619 20:27:12.678066 22874 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:12.678135 22874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:12.678279 22874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:12.679301 22874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "9ed49ef081d14565b2b1f9b633561bdf"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "8d144b12224aa53c5eec9e3c3e4465b4"
server_key_iv: "34c84b5021972687b36b4b1a423f6286"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.679735 22874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "9ed49ef081d14565b2b1f9b633561bdf"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "8d144b12224aa53c5eec9e3c3e4465b4"
server_key_iv: "34c84b5021972687b36b4b1a423f6286"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.680974 22874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:12.681663 22889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.681819 22874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:12.681915 22874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "9ed49ef081d14565b2b1f9b633561bdf"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "8d144b12224aa53c5eec9e3c3e4465b4"
server_key_iv: "34c84b5021972687b36b4b1a423f6286"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.681994 22874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:12.707921 22874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:12.709776 22874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:12.709945 22874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:12.710192 22874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:12.710515 22874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:12.710583 22874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.710631 22874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:12.710667 22874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.716423 22874 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:41967
I20260619 20:27:12.716468 23002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:41967 every 8 connection(s)
I20260619 20:27:12.716770 22874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:27:12.720197 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 22874
I20260619 20:27:12.720296 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:12.720474 22108 external_mini_cluster.cc:1428] Setting key a73e613808608f1674c6b416146e4f9e
Jun 19 20:27:12 dist-test-slave-f9rk krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900832, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:12.727226 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.717471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:52241 (local address 127.21.151.62:41015)
0619 20:27:12.717573 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:12.717595 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:12.717748 (+ 153us) server_negotiation.cc:407] Connection header received
0619 20:27:12.718683 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:12.718685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:12.718696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:12.718747 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:12.719401 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.719977 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.721329 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.721520 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.721615 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.724061 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:12.724063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:12.724063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:12.724071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:12.725810 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:12.726210 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:12.726211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:12.726212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:12.726277 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:12.726546 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:12.726547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:12.726548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:12.726699 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:12.726786 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:12.727012 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 20:27:12.727096 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 20:27:12.727262 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.717416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:41015 (local address 127.21.151.2:52241)
0619 20:27:12.717615 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:12.717659 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:12.718552 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:12.718828 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:12.718832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:12.719124 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:12.719306 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.719310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.720080 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.720081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.721238 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.721241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.721629 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.721630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.721762 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.722322 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:12.722333 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:12.723961 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:12.725905 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:12.725911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:12.725912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:12.726135 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:12.726347 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:12.726348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:12.726348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:12.726476 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:12.726778 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:12.726781 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:12.726890 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 20:27:12.726967 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:27:12.727089 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260619 20:27:12.727857 23003 heartbeater.cc:347] Connected to a master server at 127.21.151.62:41015
I20260619 20:27:12.727959 23003 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:12.728147 23003 heartbeater.cc:511] Master 127.21.151.62:41015 requested a full tablet report, sending...
I20260619 20:27:12.728561 22678 ts_manager.cc:194] Registered new tserver with Master: 9ed49ef081d14565b2b1f9b633561bdf (127.21.151.2:41967)
I20260619 20:27:12.728937 22678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:52241
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:27:12.744474 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41015
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:12.824064 23008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:12.824255 23008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:12.824278 23008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:12.825419 23008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:12.825460 23008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:12.825486 23008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:12.826982 23008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41015
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:12.827231 23008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:12.827481 23008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:12.832439 23014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:12.832449 23013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:12.832744 23008 server_base.cc:1034] running on GCE node
W20260619 20:27:12.832455 23016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:12.833026 23008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:12.833279 23008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:12.834440 23008 hybrid_clock.cc:648] HybridClock initialized: now 1781900832834435 us; error 37 us; skew 500 ppm
Jun 19 20:27:12 dist-test-slave-f9rk krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900832, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:12.837005 23008 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:27:12.837456 23008 webserver.cc:460] Webserver started at http://127.21.151.3:41949/ using document root <none> and password file <none>
I20260619 20:27:12.837684 23008 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:12.837730 23008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:12.837888 23008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:12.838922 23008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "16ecd3bbbf8948ddaea89e6777b50a96"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "0f729342313efa2b07ef4a6f9a47945f"
server_key_iv: "6d76f2f01cf197982a96091f805ffde1"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.839273 23008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "16ecd3bbbf8948ddaea89e6777b50a96"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "0f729342313efa2b07ef4a6f9a47945f"
server_key_iv: "6d76f2f01cf197982a96091f805ffde1"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.840641 23008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:27:12.841323 23023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.841503 23008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:12.841594 23008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "16ecd3bbbf8948ddaea89e6777b50a96"
format_stamp: "Formatted at 2026-06-19 20:27:12 on dist-test-slave-f9rk"
server_key: "0f729342313efa2b07ef4a6f9a47945f"
server_key_iv: "6d76f2f01cf197982a96091f805ffde1"
server_key_version: "encryptionkey@0"
I20260619 20:27:12.841665 23008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:12.851740 23008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:12.853413 23008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:12.853566 23008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:12.853778 23008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:12.854084 23008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:12.854149 23008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.854192 23008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:12.854219 23008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:12.860124 23008 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:34597
I20260619 20:27:12.860174 23136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:34597 every 8 connection(s)
I20260619 20:27:12.860484 23008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:27:12 dist-test-slave-f9rk krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900832, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:12.868803 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.861128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:35073 (local address 127.21.151.62:41015)
0619 20:27:12.861175 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:12.861178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:12.861396 (+ 218us) server_negotiation.cc:407] Connection header received
0619 20:27:12.862253 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:12.862254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:12.862264 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:12.862316 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:12.862868 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.863348 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.864303 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.864482 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.864534 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.866512 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:12.866513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:12.866514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:12.866520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:12.867677 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:12.868096 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:12.868097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:12.868097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:12.868144 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:12.868337 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:12.868338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:12.868338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:12.868429 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:12.868476 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:12.868686 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 20:27:12.868740 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:12.869025 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.861124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:41015 (local address 127.21.151.3:35073)
0619 20:27:12.861276 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:12.861310 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:12.862171 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:12.862369 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:12.862372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:12.862619 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:12.862801 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.862805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.863421 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.863421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.864230 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.864232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.864551 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.864551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.864650 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.865157 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:12.865165 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:12.866429 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:12.867741 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:12.867746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:12.867748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:12.868030 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:12.868197 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:12.868198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:12.868198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:12.868288 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:12.868480 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:12.868484 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:12.868585 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 20:27:12.868661 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:27:12.868768 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260619 20:27:12.869103 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 23008
I20260619 20:27:12.869190 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:12.869380 22108 external_mini_cluster.cc:1428] Setting key 2558b9681b14d0012dc56045b06dbe75
I20260619 20:27:12.869585 23137 heartbeater.cc:347] Connected to a master server at 127.21.151.62:41015
I20260619 20:27:12.869689 23137 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:12.869880 23137 heartbeater.cc:511] Master 127.21.151.62:41015 requested a full tablet report, sending...
I20260619 20:27:12.870235 22678 ts_manager.cc:194] Registered new tserver with Master: 16ecd3bbbf8948ddaea89e6777b50a96 (127.21.151.3:34597)
I20260619 20:27:12.870647 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:27:12.870767 22678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:35073
I20260619 20:27:12.877882 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.872419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.21.151.62:41015)
0619 20:27:12.872456 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:12.872458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:12.872582 (+ 124us) server_negotiation.cc:407] Connection header received
0619 20:27:12.872641 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:12.872642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:12.872666 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:12.872706 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:12.873302 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.873736 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.874380 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.874566 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.874612 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.875681 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:12.875682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:12.875682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:12.875688 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:12.877035 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:12.877319 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:12.877320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:12.877320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:12.877368 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:12.877509 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:12.877510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:12.877510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:12.877623 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:12.877688 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:12.877776 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 20:27:12.877827 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:27:12.879827 22678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:12.881353 22678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:12.887763 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.883442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.21.151.2:41967)
0619 20:27:12.883596 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:12.883601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:12.883610 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:12.883678 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:12.883680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:12.883696 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:12.883987 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:12.884085 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.885259 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.886588 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.887306 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.887399 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.887460 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:12.887591 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 20:27:12.887663 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260619 20:27:12.887754 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.883541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:34597 (local address 127.0.0.1:38462)
0619 20:27:12.883744 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:12.883755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:12.883797 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:12.884100 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:12.884101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:12.884103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:12.884148 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.884150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.885194 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.885195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.886623 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.886626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.887491 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.887492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.887559 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.887563 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:12.887596 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 20:27:12.887660 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":55,"thread_start_us":35,"threads_started":1}
I20260619 20:27:12.887760 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.883340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:41967 (local address 127.0.0.1:53344)
0619 20:27:12.883446 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:12.883484 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:12.883540 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:12.883813 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:12.883816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:12.883818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:12.883886 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.883888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.885355 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.885356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.886419 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.886421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.887399 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.887400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.887484 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.887500 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 20:27:12.887569 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:27:12.887644 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":64,"threads_started":1}
I20260619 20:27:12.887902 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.883696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:40921 (local address 127.0.0.1:35774)
0619 20:27:12.884049 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:12.884059 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:12.884109 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:12.884465 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:12.884466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:12.884468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:12.884509 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.884510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.885673 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.885674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.886832 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:12.886834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.887598 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.887599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:12.887671 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.887675 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:12.887725 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:27:12.887758 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":49,"threads_started":1}
I20260619 20:27:12.887918 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.883811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35774 (local address 127.21.151.1:40921)
0619 20:27:12.884004 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:12.884009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:12.884165 (+ 156us) server_negotiation.cc:407] Connection header received
0619 20:27:12.884204 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:12.884206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:12.884220 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:12.884386 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:12.884579 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.885582 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.886908 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.887437 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.887501 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.887555 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:12.887797 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 20:27:12.887844 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260619 20:27:12.888322 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:12.883799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.21.151.3:34597)
0619 20:27:12.883933 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:12.883936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:12.883945 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:12.884004 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:12.884006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:12.884016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:12.884102 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:12.884203 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.885095 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.886706 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:12.887397 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:12.888122 (+ 725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:12.888184 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:12.888208 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 20:27:12.888248 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260619 20:27:12.888689 22803 tablet_service.cc:1467] Processing CreateTablet for tablet b3be1ba1beba4372809accabbbd85177 (DEFAULT_TABLE table=test-table [id=1eaed739253848ff88152d6cad87fc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:12.888931 23071 tablet_service.cc:1467] Processing CreateTablet for tablet b3be1ba1beba4372809accabbbd85177 (DEFAULT_TABLE table=test-table [id=1eaed739253848ff88152d6cad87fc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:12.889063 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3be1ba1beba4372809accabbbd85177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:12.889223 23071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3be1ba1beba4372809accabbbd85177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:12.892102 22937 tablet_service.cc:1467] Processing CreateTablet for tablet b3be1ba1beba4372809accabbbd85177 (DEFAULT_TABLE table=test-table [id=1eaed739253848ff88152d6cad87fc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:12.892385 23151 tablet_bootstrap.cc:492] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308: Bootstrap starting.
I20260619 20:27:12.892426 22937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3be1ba1beba4372809accabbbd85177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:12.893018 23151 tablet_bootstrap.cc:654] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:12.893550 23151 log.cc:824] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:12.893875 23152 tablet_bootstrap.cc:492] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96: Bootstrap starting.
I20260619 20:27:12.894250 23151 tablet_bootstrap.cc:492] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308: No bootstrap required, opened a new log
I20260619 20:27:12.894331 23151 ts_tablet_manager.cc:1397] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:12.894507 23152 tablet_bootstrap.cc:654] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:12.895064 23152 log.cc:824] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:12.895766 23155 tablet_bootstrap.cc:492] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf: Bootstrap starting.
I20260619 20:27:12.896014 23152 tablet_bootstrap.cc:492] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96: No bootstrap required, opened a new log
I20260619 20:27:12.896087 23152 ts_tablet_manager.cc:1397] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:12.896294 23151 raft_consensus.cc:348] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed49ef081d14565b2b1f9b633561bdf" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 41967 } } peers { permanent_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34597 } } peers { permanent_uuid: "453a796931fa4de2ba0a5d02b73ad308" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40921 } }
I20260619 20:27:12.896477 23155 tablet_bootstrap.cc:654] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:12.896478 23151 raft_consensus.cc:374] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:12.896515 23151 raft_consensus.cc:729] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453a796931fa4de2ba0a5d02b73ad308, State: Initialized, Role: FOLLOWER
I20260619 20:27:12.896646 23151 consensus_queue.cc:260] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed49ef081d14565b2b1f9b633561bdf" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 41967 } } peers { permanent_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34597 } } peers { permanent_uuid: "453a796931fa4de2ba0a5d02b73ad308" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40921 } }
I20260619 20:27:12.896960 23151 ts_tablet_manager.cc:1428] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:27:12.896963 22869 heartbeater.cc:503] Master 127.21.151.62:41015 was elected leader, sending a full tablet report...
I20260619 20:27:12.897020 23155 log.cc:824] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:12.897750 23155 tablet_bootstrap.cc:492] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf: No bootstrap required, opened a new log
I20260619 20:27:12.897814 23155 ts_tablet_manager.cc:1397] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:12.897985 23152 raft_consensus.cc:348] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed49ef081d14565b2b1f9b633561bdf" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 41967 } } peers { permanent_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34597 } } peers { permanent_uuid: "453a796931fa4de2ba0a5d02b73ad308" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40921 } }
I20260619 20:27:12.898141 23152 raft_consensus.cc:374] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:12.898183 23152 raft_consensus.cc:729] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ecd3bbbf8948ddaea89e6777b50a96, State: Initialized, Role: FOLLOWER
I20260619 20:27:12.898298 23152 consensus_queue.cc:260] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed49ef081d14565b2b1f9b633561bdf" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 41967 } } peers { permanent_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34597 } } peers { permanent_uuid: "453a796931fa4de2ba0a5d02b73ad308" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40921 } }
I20260619 20:27:12.898566 23137 heartbeater.cc:503] Master 127.21.151.62:41015 was elected leader, sending a full tablet report...
I20260619 20:27:12.898815 23152 ts_tablet_manager.cc:1428] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:27:12.899730 23155 raft_consensus.cc:348] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed49ef081d14565b2b1f9b633561bdf" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 41967 } } peers { permanent_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34597 } } peers { permanent_uuid: "453a796931fa4de2ba0a5d02b73ad308" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40921 } }
I20260619 20:27:12.899853 23155 raft_consensus.cc:374] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:12.899875 23155 raft_consensus.cc:729] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed49ef081d14565b2b1f9b633561bdf, State: Initialized, Role: FOLLOWER
I20260619 20:27:12.900002 23155 consensus_queue.cc:260] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed49ef081d14565b2b1f9b633561bdf" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 41967 } } peers { permanent_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34597 } } peers { permanent_uuid: "453a796931fa4de2ba0a5d02b73ad308" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40921 } }
I20260619 20:27:12.900265 23155 ts_tablet_manager.cc:1428] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:27:12.900288 23003 heartbeater.cc:503] Master 127.21.151.62:41015 was elected leader, sending a full tablet report...
W20260619 20:27:12.967379 23004 tablet.cc:2369] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:27:13.055336 22870 tablet.cc:2369] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:27:13.056020 23156 raft_consensus.cc:482] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:13.056165 23156 raft_consensus.cc:504] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed49ef081d14565b2b1f9b633561bdf" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 41967 } } peers { permanent_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34597 } } peers { permanent_uuid: "453a796931fa4de2ba0a5d02b73ad308" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40921 } }
I20260619 20:27:13.056515 23156 leader_election.cc:288] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ed49ef081d14565b2b1f9b633561bdf (127.21.151.2:41967), 16ecd3bbbf8948ddaea89e6777b50a96 (127.21.151.3:34597)
I20260619 20:27:13.059577 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:13.056695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:41967 (local address 127.21.151.1:53945)
0619 20:27:13.056759 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:13.056773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:13.056857 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:13.057040 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:13.057041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:13.057042 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:13.057130 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:13.057132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.057909 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.057910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:13.058761 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:13.058763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.059312 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.059313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:13.059395 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:13.059399 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:13.059441 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 20:27:13.059468 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":61}
I20260619 20:27:13.059582 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:13.056783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:53945 (local address 127.21.151.2:41967)
0619 20:27:13.056828 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:13.056830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:13.056858 (+ 28us) server_negotiation.cc:407] Connection header received
0619 20:27:13.056955 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:13.056957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:13.056970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:13.057035 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:13.057220 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.057825 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.058862 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.059230 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.059285 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:13.059315 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:13.059471 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:27:13.059513 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:27:13.059653 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:13.056814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:34597 (local address 127.21.151.1:56097)
0619 20:27:13.056961 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:13.056972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:13.057026 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:13.057218 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:13.057219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:13.057221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:13.057308 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:13.057310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.057993 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.057994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:13.058762 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:13.058764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.059345 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.059346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:13.059415 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:13.059419 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:13.059494 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:27:13.059529 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260619 20:27:13.059650 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:13.056937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:56097 (local address 127.21.151.3:34597)
0619 20:27:13.056982 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:13.056984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:13.057013 (+ 29us) server_negotiation.cc:407] Connection header received
0619 20:27:13.057101 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:13.057102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:13.057114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:13.057176 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:13.057397 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.057909 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.058865 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.059238 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.059325 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:13.059332 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:13.059517 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 20:27:13.059556 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:27:13.059994 23091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3be1ba1beba4372809accabbbd85177" candidate_uuid: "453a796931fa4de2ba0a5d02b73ad308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" is_pre_election: true
I20260619 20:27:13.060073 22957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3be1ba1beba4372809accabbbd85177" candidate_uuid: "453a796931fa4de2ba0a5d02b73ad308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed49ef081d14565b2b1f9b633561bdf" is_pre_election: true
I20260619 20:27:13.060168 23091 raft_consensus.cc:2454] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453a796931fa4de2ba0a5d02b73ad308 in term 0.
I20260619 20:27:13.060241 22957 raft_consensus.cc:2454] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453a796931fa4de2ba0a5d02b73ad308 in term 0.
I20260619 20:27:13.060427 22757 leader_election.cc:302] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ecd3bbbf8948ddaea89e6777b50a96, 453a796931fa4de2ba0a5d02b73ad308; no voters:
I20260619 20:27:13.060573 23156 raft_consensus.cc:2781] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:13.060657 23156 raft_consensus.cc:482] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:13.060683 23156 raft_consensus.cc:3037] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:13.061378 23156 raft_consensus.cc:504] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed49ef081d14565b2b1f9b633561bdf" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 41967 } } peers { permanent_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34597 } } peers { permanent_uuid: "453a796931fa4de2ba0a5d02b73ad308" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40921 } }
I20260619 20:27:13.061514 23156 leader_election.cc:288] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [CANDIDATE]: Term 1 election: Requested vote from peers 9ed49ef081d14565b2b1f9b633561bdf (127.21.151.2:41967), 16ecd3bbbf8948ddaea89e6777b50a96 (127.21.151.3:34597)
I20260619 20:27:13.061651 22957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3be1ba1beba4372809accabbbd85177" candidate_uuid: "453a796931fa4de2ba0a5d02b73ad308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed49ef081d14565b2b1f9b633561bdf"
I20260619 20:27:13.061694 23091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3be1ba1beba4372809accabbbd85177" candidate_uuid: "453a796931fa4de2ba0a5d02b73ad308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ecd3bbbf8948ddaea89e6777b50a96"
I20260619 20:27:13.061733 22957 raft_consensus.cc:3037] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:13.061784 23091 raft_consensus.cc:3037] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:13.062428 22957 raft_consensus.cc:2454] T b3be1ba1beba4372809accabbbd85177 P 9ed49ef081d14565b2b1f9b633561bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453a796931fa4de2ba0a5d02b73ad308 in term 1.
I20260619 20:27:13.062461 23091 raft_consensus.cc:2454] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453a796931fa4de2ba0a5d02b73ad308 in term 1.
I20260619 20:27:13.062590 22757 leader_election.cc:302] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453a796931fa4de2ba0a5d02b73ad308, 9ed49ef081d14565b2b1f9b633561bdf; no voters:
I20260619 20:27:13.062692 23156 raft_consensus.cc:2781] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:13.062853 23156 raft_consensus.cc:686] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [term 1 LEADER]: Becoming Leader. State: Replica: 453a796931fa4de2ba0a5d02b73ad308, State: Running, Role: LEADER
I20260619 20:27:13.062952 23156 consensus_queue.cc:237] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed49ef081d14565b2b1f9b633561bdf" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 41967 } } peers { permanent_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34597 } } peers { permanent_uuid: "453a796931fa4de2ba0a5d02b73ad308" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40921 } }
I20260619 20:27:13.063782 22678 catalog_manager.cc:5495] T b3be1ba1beba4372809accabbbd85177 P 453a796931fa4de2ba0a5d02b73ad308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 453a796931fa4de2ba0a5d02b73ad308 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "453a796931fa4de2ba0a5d02b73ad308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed49ef081d14565b2b1f9b633561bdf" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 41967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ecd3bbbf8948ddaea89e6777b50a96" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "453a796931fa4de2ba0a5d02b73ad308" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40921 } health_report { overall_health: HEALTHY } } }
W20260619 20:27:13.111006 23138 tablet.cc:2369] T b3be1ba1beba4372809accabbbd85177 P 16ecd3bbbf8948ddaea89e6777b50a96: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 20:27:13 dist-test-slave-f9rk krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.1@KRBTEST.COM
I20260619 20:27:13.120240 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:13.110328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.21.151.1:40921)
0619 20:27:13.110412 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:13.110414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:13.110504 (+ 90us) server_negotiation.cc:407] Connection header received
0619 20:27:13.110597 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:13.110598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:13.110609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:13.110655 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:13.111222 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.111730 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.112374 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.112545 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.112599 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:13.117580 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:13.117601 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:13.117601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:13.117608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:13.119225 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:13.119512 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:13.119514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:13.119515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:13.119568 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:13.119771 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:13.119771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:13.119772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:13.119907 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:13.120008 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:13.120070 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 20:27:13.120133 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 20:27:13 dist-test-slave-f9rk krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:27:13 dist-test-slave-f9rk krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.151.1@KRBTEST.COM
I20260619 20:27:13.142551 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:13.135458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.21.151.1:40921)
0619 20:27:13.135523 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:13.135526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:13.135609 (+ 83us) server_negotiation.cc:407] Connection header received
0619 20:27:13.135761 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:13.135762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:13.135775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:13.135846 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:13.136410 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.136922 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.137699 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.137906 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.138006 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:13.140150 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:13.140151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:13.140152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:13.140159 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:13.141322 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:13.141697 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:13.141698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:13.141699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:13.141759 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:13.141954 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:13.141955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:13.141955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:13.142069 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:13.142177 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:13.142363 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 20:27:13.142445 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 20:27:13 dist-test-slave-f9rk krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:27:13 dist-test-slave-f9rk krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.1@KRBTEST.COM
I20260619 20:27:13.164542 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:13.157658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.21.151.1:40921)
0619 20:27:13.157725 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:13.157727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:13.157806 (+ 79us) server_negotiation.cc:407] Connection header received
0619 20:27:13.157910 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:13.157911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:13.157921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:13.157995 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:13.158540 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.159026 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.159821 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:13.160020 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:13.160108 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:13.162211 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:13.162212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:13.162212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:13.162220 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:13.163366 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:13.163766 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:13.163767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:13.163768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:13.163828 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:13.164034 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:13.164035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:13.164035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:13.164137 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:13.164238 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:13.164344 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 20:27:13.164423 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 20:27:13.165045 22783 tablet_service.cc:2908] Rejecting scan request for tablet b3be1ba1beba4372809accabbbd85177: Uninitialized: clean time has not yet been initialized
I20260619 20:27:13.165920 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 22740
I20260619 20:27:13.170710 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 22874
I20260619 20:27:13.175132 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 23008
I20260619 20:27:13.179347 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 22646
2026-06-19T20:27:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2074 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:13 dist-test-slave-f9rk krb5kdc[23175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:13 dist-test-slave-f9rk krb5kdc[23175](info): set up 2 sockets
Jun 19 20:27:13 dist-test-slave-f9rk krb5kdc[23175](info): commencing operation
krb5kdc: starting...
W20260619 20:27:14.229108 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:14 dist-test-slave-f9rk krb5kdc[23175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:14.344270 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:38703
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:38703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:14.423573 23188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:14.423859 23188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:14.423882 23188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:14.425053 23188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:14.425099 23188 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 20:27:14.425114 23188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:14.425153 23188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:14.425182 23188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:14.425199 23188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:14.426645 23188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:38703
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:38703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:14.426892 23188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:14.427122 23188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:14.431800 23193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:14.431813 23188 server_base.cc:1034] running on GCE node
W20260619 20:27:14.431777 23194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:14.431782 23196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:14.432258 23188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:14.432493 23188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:14.433665 23188 hybrid_clock.cc:648] HybridClock initialized: now 1781900834433643 us; error 37 us; skew 500 ppm
Jun 19 20:27:14 dist-test-slave-f9rk krb5kdc[23175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900834, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:14.436256 23188 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:14.436668 23188 webserver.cc:460] Webserver started at http://127.21.151.62:44883/ using document root <none> and password file <none>
I20260619 20:27:14.436897 23188 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:14.436939 23188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:14.437124 23188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:14.438189 23188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "26cd84fdae4f463391a9d330567c8dab"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "ec770ba92524db36d8a6980f379cf972"
server_key_iv: "78dab009ccd22756cf18dae939e3712d"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.438545 23188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "26cd84fdae4f463391a9d330567c8dab"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "ec770ba92524db36d8a6980f379cf972"
server_key_iv: "78dab009ccd22756cf18dae939e3712d"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.439908 23188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:27:14.440622 23203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.440809 23188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.440898 23188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "26cd84fdae4f463391a9d330567c8dab"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "ec770ba92524db36d8a6980f379cf972"
server_key_iv: "78dab009ccd22756cf18dae939e3712d"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.440970 23188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:14.471164 23188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:14.473587 23188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:14.473697 23188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:14.478582 23188 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:38703
I20260619 20:27:14.478677 23265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:38703 every 8 connection(s)
I20260619 20:27:14.478996 23188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:14.479512 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 23188
I20260619 20:27:14.479583 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:14.479789 22108 external_mini_cluster.cc:1428] Setting key c65d21830f0ef11cf28cb2251db6d358
I20260619 20:27:14.479792 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:14.482334 23266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab: Bootstrap starting.
I20260619 20:27:14.482936 23266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:14.483223 23266 log.cc:824] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:14.484025 23266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab: No bootstrap required, opened a new log
I20260619 20:27:14.485877 23266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cd84fdae4f463391a9d330567c8dab" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38703 } }
I20260619 20:27:14.486009 23266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:14.486032 23266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cd84fdae4f463391a9d330567c8dab, State: Initialized, Role: FOLLOWER
I20260619 20:27:14.486164 23266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cd84fdae4f463391a9d330567c8dab" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38703 } }
I20260619 20:27:14.486253 23266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Jun 19 20:27:14 dist-test-slave-f9rk krb5kdc[23175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:14.486307 23266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:14.486356 23266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:14.486992 23266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cd84fdae4f463391a9d330567c8dab" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38703 } }
I20260619 20:27:14.487108 23266 leader_election.cc:302] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26cd84fdae4f463391a9d330567c8dab; no voters:
I20260619 20:27:14.487270 23266 leader_election.cc:288] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:14.487327 23271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:14.487486 23271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [term 1 LEADER]: Becoming Leader. State: Replica: 26cd84fdae4f463391a9d330567c8dab, State: Running, Role: LEADER
I20260619 20:27:14.487645 23266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:14.487670 23271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cd84fdae4f463391a9d330567c8dab" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38703 } }
I20260619 20:27:14.488296 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26cd84fdae4f463391a9d330567c8dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cd84fdae4f463391a9d330567c8dab" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38703 } } }
I20260619 20:27:14.488458 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:14.488875 23280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:14.489430 23280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:14.490459 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26cd84fdae4f463391a9d330567c8dab. Latest consensus state: current_term: 1 leader_uuid: "26cd84fdae4f463391a9d330567c8dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cd84fdae4f463391a9d330567c8dab" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38703 } } }
I20260619 20:27:14.490612 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:14.491019 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.480336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.21.151.62:38703)
0619 20:27:14.480489 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:14.480498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:14.480530 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:27:14.480995 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:14.481001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:14.481047 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:14.481197 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:14.481812 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.482739 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.483612 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.484273 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.484376 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.486784 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:14.486788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:14.486788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:14.486800 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:14.489248 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.489662 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.489665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.489665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.489764 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.489963 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.489964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.489964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.490275 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:14.490432 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:14.490773 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 20:27:14.490884 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260619 20:27:14.496008 23280 catalog_manager.cc:1347] Generated new cluster ID: c9e851312e06466ca68c6117c2493514
I20260619 20:27:14.496079 23280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:14.513260 23280 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:14.513955 23280 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:14.516842 23280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab: Generated new TSK 0
I20260619 20:27:14.517016 23280 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:27:14.540191 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38703
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:14.619521 23292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:14.619732 23292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:14.619756 23292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:14.620944 23292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:14.620991 23292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:14.621037 23292 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:27:14.621073 23292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:14.622469 23292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.151.62:38703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:14.622720 23292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:14.622951 23292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:14.627696 23298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:14.627714 23297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:14.627697 23300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:14.628002 23292 server_base.cc:1034] running on GCE node
I20260619 20:27:14.628207 23292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:14.628435 23292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:14.629575 23292 hybrid_clock.cc:648] HybridClock initialized: now 1781900834629565 us; error 27 us; skew 500 ppm
Jun 19 20:27:14 dist-test-slave-f9rk krb5kdc[23175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900834, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:14.632103 23292 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:27:14.632509 23292 webserver.cc:460] Webserver started at http://127.21.151.1:41089/ using document root <none> and password file <none>
I20260619 20:27:14.632748 23292 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:14.632818 23292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:14.632951 23292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:14.633967 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "e5c17f39faf346cd9bfca86919f7d5bf"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "1a77a0bcdcf87224fcb2f3564a32a869"
server_key_iv: "5fc353e0857b43dd51d3c9347aa1186b"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.634438 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "e5c17f39faf346cd9bfca86919f7d5bf"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "1a77a0bcdcf87224fcb2f3564a32a869"
server_key_iv: "5fc353e0857b43dd51d3c9347aa1186b"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.635738 23292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:14.636428 23307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.636595 23292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:14.636685 23292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "e5c17f39faf346cd9bfca86919f7d5bf"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "1a77a0bcdcf87224fcb2f3564a32a869"
server_key_iv: "5fc353e0857b43dd51d3c9347aa1186b"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.636768 23292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:14.657255 23292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:14.659106 23292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:14.659276 23292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:14.659848 23292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:14.659909 23292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.659942 23292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:14.659972 23292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.666646 23292 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:40925
I20260619 20:27:14.666721 23422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:40925 every 8 connection(s)
Jun 19 20:27:14 dist-test-slave-f9rk krb5kdc[23175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900834, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:14.667083 23292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:27:14.670133 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.660310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:48095 (local address 127.21.151.62:38703)
0619 20:27:14.660391 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:14.660394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:14.660561 (+ 167us) server_negotiation.cc:407] Connection header received
0619 20:27:14.661065 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:14.661067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:14.661081 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:14.661167 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:14.662057 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.662887 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.664470 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.664822 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.664906 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.667471 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:14.667473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:14.667473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:14.667480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:14.669058 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.669433 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.669433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.669434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.669484 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.669644 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.669644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.669644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.669734 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:14.669777 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:14.670007 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 20:27:14.670070 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 20:27:14.670239 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.660286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38703 (local address 127.21.151.1:48095)
0619 20:27:14.660427 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:14.660467 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:14.660958 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:14.661252 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:14.661256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:14.661656 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:14.661939 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.661948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.662997 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.662998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.664371 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.664374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.664910 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.664911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.665033 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.665513 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:14.665530 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:14.667362 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:14.669133 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.669138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.669139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:14.669366 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:14.669525 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.669525 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.669526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:14.669608 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:14.669768 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:14.669771 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:14.669869 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 20:27:14.669943 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:27:14.670057 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260619 20:27:14.670691 23423 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38703
I20260619 20:27:14.670774 23423 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:14.670957 23423 heartbeater.cc:511] Master 127.21.151.62:38703 requested a full tablet report, sending...
I20260619 20:27:14.671609 23219 ts_manager.cc:194] Registered new tserver with Master: e5c17f39faf346cd9bfca86919f7d5bf (127.21.151.1:40925)
I20260619 20:27:14.672490 23219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:48095
I20260619 20:27:14.674966 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 23292
I20260619 20:27:14.675086 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:14.675261 22108 external_mini_cluster.cc:1428] Setting key 305d8a96f6d2580ed698d97c60188243
I20260619 20:27:14.677199 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.671906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:53911 (local address 127.21.151.62:38703)
0619 20:27:14.672090 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:14.672092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:14.672099 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:27:14.672118 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:14.672119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:14.672127 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:14.672184 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:14.672456 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.673061 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.673740 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.673907 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.673953 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.674663 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:14.674664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:14.674664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:14.674669 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:14.676077 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.676424 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.676425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.676425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.676477 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.676676 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.676677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.676677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.676781 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:14.676852 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:14.677026 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 20:27:14.677113 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260619 20:27:14.677211 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.671736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38703 (local address 127.21.151.1:53911)
0619 20:27:14.671873 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:14.671885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:14.671938 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:14.672192 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:14.672193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:14.672344 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:14.672389 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.672390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.673155 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.673156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.673672 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.673673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.673965 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.673966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.674028 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.674342 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:14.674345 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:14.674592 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:14.676177 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.676178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.676178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:14.676341 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:14.676550 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.676550 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.676551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:14.676607 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:14.676846 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:14.676846 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:14.676889 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 20:27:14.676964 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:27:14.677081 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:27:14.698387 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38703
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:14.777441 23432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:14.777673 23432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:14.777709 23432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:14.778857 23432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:14.778932 23432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:14.778970 23432 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:27:14.779006 23432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:14.780478 23432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.151.62:38703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:14.780791 23432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:14.781028 23432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:14.785893 23437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:14.785872 23438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:14.785888 23440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:14.786113 23432 server_base.cc:1034] running on GCE node
I20260619 20:27:14.786397 23432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:14.786628 23432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:14.787779 23432 hybrid_clock.cc:648] HybridClock initialized: now 1781900834787773 us; error 51 us; skew 500 ppm
Jun 19 20:27:14 dist-test-slave-f9rk krb5kdc[23175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900834, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:14.790253 23432 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:27:14.790630 23432 webserver.cc:460] Webserver started at http://127.21.151.2:37377/ using document root <none> and password file <none>
I20260619 20:27:14.790874 23432 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:14.790944 23432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:14.791079 23432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:14.792266 23432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "14ff040631cc45fba1fd8c457bfdfd8e"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "edb6401b1804f097a9d5a00a4188e231"
server_key_iv: "d1403f8d3aa30abd3a5cb1d34525e521"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.792634 23432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "14ff040631cc45fba1fd8c457bfdfd8e"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "edb6401b1804f097a9d5a00a4188e231"
server_key_iv: "d1403f8d3aa30abd3a5cb1d34525e521"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.793959 23432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:14.794695 23447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.794947 23432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:14.795058 23432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "14ff040631cc45fba1fd8c457bfdfd8e"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "edb6401b1804f097a9d5a00a4188e231"
server_key_iv: "d1403f8d3aa30abd3a5cb1d34525e521"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.795146 23432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:14.813215 23432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:14.815479 23432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:14.815737 23432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:14.816252 23432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:14.816314 23432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.816360 23432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:14.816388 23432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.823314 23432 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:33987
I20260619 20:27:14.823371 23562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:33987 every 8 connection(s)
Jun 19 20:27:14 dist-test-slave-f9rk krb5kdc[23175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900834, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:14.823764 23432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:27:14.826952 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.816793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:57933 (local address 127.21.151.62:38703)
0619 20:27:14.816836 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:14.816838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:14.817066 (+ 228us) server_negotiation.cc:407] Connection header received
0619 20:27:14.817563 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:14.817565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:14.817578 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:14.817632 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:14.818606 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.819454 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.821160 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.821410 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.821466 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.824246 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:14.824247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:14.824247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:14.824255 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:14.825774 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.826128 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.826129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.826130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.826183 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.826384 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.826384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.826385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.826480 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:14.826528 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:14.826788 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 20:27:14.826862 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:14.827025 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.816784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38703 (local address 127.21.151.2:57933)
0619 20:27:14.816933 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:14.816966 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:14.817447 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:14.817776 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:14.817781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:14.818170 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:14.818481 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.818488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.819559 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.819560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.821069 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.821072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.821490 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.821492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.821674 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.822273 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:14.822283 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:14.824122 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:14.825850 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.825856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.825857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:14.826063 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:14.826241 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.826242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.826242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:14.826332 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:14.826532 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:14.826535 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:14.826655 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 20:27:14.826753 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 20:27:14.826879 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
I20260619 20:27:14.827651 23563 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38703
I20260619 20:27:14.827740 23563 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:14.827915 23563 heartbeater.cc:511] Master 127.21.151.62:38703 requested a full tablet report, sending...
I20260619 20:27:14.828316 23219 ts_manager.cc:194] Registered new tserver with Master: 14ff040631cc45fba1fd8c457bfdfd8e (127.21.151.2:33987)
I20260619 20:27:14.828806 23219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:57933
I20260619 20:27:14.833215 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 23432
I20260619 20:27:14.833294 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:14.833506 22108 external_mini_cluster.cc:1428] Setting key c79c6a31322edabd83ff8a206ba2c81b
I20260619 20:27:14.834878 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.828705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:50975 (local address 127.21.151.62:38703)
0619 20:27:14.828749 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:14.828751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:14.828837 (+ 86us) server_negotiation.cc:407] Connection header received
0619 20:27:14.828979 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:14.828979 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:14.828989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:14.829033 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:14.829406 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.829904 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.830629 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.830838 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.830935 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.831942 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:14.831944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:14.831944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:14.831951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:14.833560 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.834059 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.834059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.834060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.834112 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.834315 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.834316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.834316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.834468 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:14.834551 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:14.834719 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 20:27:14.834787 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:27:14.834908 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.828648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38703 (local address 127.21.151.2:50975)
0619 20:27:14.828772 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:14.828783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:14.828846 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:14.829056 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:14.829056 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:14.829264 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:14.829316 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.829317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.830011 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.830012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.830527 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.830529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.830937 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.830938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.831009 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.831476 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:14.831481 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:14.831828 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:14.833643 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.833643 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.833644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:14.833908 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:14.834182 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.834183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.834183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:14.834244 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:14.834544 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:14.834545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:14.834593 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 20:27:14.834678 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:27:14.834787 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:27:14.856488 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38703
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:44737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:14.937793 23572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:14.937997 23572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:14.938020 23572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:14.939205 23572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:14.939246 23572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:14.939266 23572 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:27:14.939311 23572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:14.940805 23572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:44737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.151.62:38703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:14.941054 23572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:14.941286 23572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:14.946331 23580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:14.946348 23577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:14.946482 23572 server_base.cc:1034] running on GCE node
W20260619 20:27:14.946391 23578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:14.946758 23572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:14.947033 23572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:14.948197 23572 hybrid_clock.cc:648] HybridClock initialized: now 1781900834948189 us; error 29 us; skew 500 ppm
Jun 19 20:27:14 dist-test-slave-f9rk krb5kdc[23175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900834, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:14.950757 23572 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:27:14.951190 23572 webserver.cc:460] Webserver started at http://127.21.151.3:46325/ using document root <none> and password file <none>
I20260619 20:27:14.951416 23572 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:14.951480 23572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:14.951648 23572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:14.952661 23572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "a5113a8f802c448496dc15417afc74d6"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "d1722ded2c1f6f11f529bbf50d83cc64"
server_key_iv: "8f4fec5caf00bbb06b2419b8091d7b60"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.953023 23572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "a5113a8f802c448496dc15417afc74d6"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "d1722ded2c1f6f11f529bbf50d83cc64"
server_key_iv: "8f4fec5caf00bbb06b2419b8091d7b60"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.954315 23572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:14.955008 23587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.955191 23572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:14.955279 23572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "a5113a8f802c448496dc15417afc74d6"
format_stamp: "Formatted at 2026-06-19 20:27:14 on dist-test-slave-f9rk"
server_key: "d1722ded2c1f6f11f529bbf50d83cc64"
server_key_iv: "8f4fec5caf00bbb06b2419b8091d7b60"
server_key_version: "encryptionkey@0"
I20260619 20:27:14.955358 23572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:14.979509 23572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:14.982048 23572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:14.982221 23572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:14.982733 23572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:14.982795 23572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.982859 23572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:14.982890 23572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:14.989820 23572 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:36657
I20260619 20:27:14.989881 23702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:36657 every 8 connection(s)
I20260619 20:27:14.990183 23572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:27:14 dist-test-slave-f9rk krb5kdc[23175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900834, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:14.991178 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 23572
I20260619 20:27:14.991277 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:14.991449 22108 external_mini_cluster.cc:1428] Setting key fb5807c70635453bdf0391df27a9e64e
I20260619 20:27:14.993912 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.983327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:58081 (local address 127.21.151.62:38703)
0619 20:27:14.983386 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:14.983388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:14.983630 (+ 242us) server_negotiation.cc:407] Connection header received
0619 20:27:14.984191 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:14.984193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:14.984205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:14.984270 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:14.985062 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.985883 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.987467 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.987852 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.987947 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.990825 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:14.990827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:14.990827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:14.990835 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:14.992530 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.992941 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.992943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.992943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.993000 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:14.993267 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:14.993268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:14.993268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:14.993390 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:14.993458 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:14.993739 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 20:27:14.993821 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:27:14.994110 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.983319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38703 (local address 127.21.151.3:58081)
0619 20:27:14.983479 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:14.983516 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:14.984082 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:14.984350 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:14.984353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:14.984667 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:14.984962 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.984968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.985972 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.985972 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.987346 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.987350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.988021 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.988022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.988232 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.988816 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:14.988825 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:14.990722 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:14.992623 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.992630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.992632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:14.992851 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:14.993073 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.993074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.993075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:14.993194 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:14.993460 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:14.993467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:14.993609 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 20:27:14.993718 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 20:27:14.993878 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260619 20:27:14.994576 23703 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38703
I20260619 20:27:14.994663 23703 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:14.994856 23703 heartbeater.cc:511] Master 127.21.151.62:38703 requested a full tablet report, sending...
I20260619 20:27:14.995208 23220 ts_manager.cc:194] Registered new tserver with Master: a5113a8f802c448496dc15417afc74d6 (127.21.151.3:36657)
I20260619 20:27:14.995739 23220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:58081
I20260619 20:27:15.000768 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.995699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:56879 (local address 127.21.151.62:38703)
0619 20:27:14.995735 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:14.995737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:14.995871 (+ 134us) server_negotiation.cc:407] Connection header received
0619 20:27:14.995954 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:14.995955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:14.995962 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:14.995999 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:14.996307 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.996841 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.997496 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.997673 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.997734 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.998645 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:14.998646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:14.998646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:14.998651 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:14.999842 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:15.000135 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:15.000136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:15.000136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:15.000182 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:15.000336 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:15.000337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:15.000337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:15.000424 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:15.000472 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.000657 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.000711 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:27:15.000778 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:14.995644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38703 (local address 127.21.151.3:56879)
0619 20:27:14.995795 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:14.995804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:14.995895 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:14.996003 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:14.996004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:14.996185 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:14.996234 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.996236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.996924 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.996924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.997418 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:14.997419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:14.997750 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:14.997751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:14.997820 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:14.998259 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:14.998263 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:14.998552 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:14.999906 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:14.999907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:14.999907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:15.000071 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:15.000237 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:15.000238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:15.000238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:15.000287 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:15.000519 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:15.000520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:15.000566 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 20:27:15.000633 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 20:27:15.000706 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260619 20:27:15.003139 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:27:15 dist-test-slave-f9rk krb5kdc[23175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900835, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:27:15 dist-test-slave-f9rk krb5kdc[23175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900835, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:15.025291 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.018627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.21.151.62:38703)
0619 20:27:15.018689 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:15.018691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:15.018699 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:15.018723 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:15.018724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:15.018732 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:15.018782 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:15.019372 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.019927 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.020740 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.020920 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.020972 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.023081 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:15.023083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:15.023083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:15.023089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:15.024360 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:15.024664 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:15.024666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:15.024666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:15.024716 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:15.024877 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:15.024877 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:15.024878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:15.024976 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:15.025051 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.025141 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.025201 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:27:15.027021 23220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:15.028796 23220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:15.035293 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.031209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.21.151.1:40925)
0619 20:27:15.031385 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:15.031391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:15.031514 (+ 123us) server_negotiation.cc:407] Connection header received
0619 20:27:15.031648 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:15.031652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:15.031670 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:15.031755 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:15.031903 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.032767 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.034232 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.034817 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.034901 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.034962 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.035074 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.035118 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260619 20:27:15.035475 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.031131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:33987 (local address 127.0.0.1:57740)
0619 20:27:15.031576 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:15.031586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:15.031658 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:15.031886 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:15.031887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:15.031889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:15.031952 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.031954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.033249 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.033250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.034264 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.034265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.035226 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.035227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.035296 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.035302 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 20:27:15.035349 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 20:27:15.035396 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":8,"thread_start_us":78,"threads_started":1}
I20260619 20:27:15.035537 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.031112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:40925 (local address 127.0.0.1:35636)
0619 20:27:15.031392 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:15.031431 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:15.031514 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:15.031758 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:15.031761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:15.031763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:15.031830 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.031832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.032857 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.032858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.034147 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.034149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.034900 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.034900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.034975 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.034988 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:27:15.035060 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 20:27:15.035109 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":69,"threads_started":1}
I20260619 20:27:15.035583 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.031346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.21.151.3:36657)
0619 20:27:15.031500 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:15.031504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:15.032059 (+ 555us) server_negotiation.cc:407] Connection header received
0619 20:27:15.032114 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:15.032117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:15.032132 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:15.032205 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:15.032322 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.033339 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.034439 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.035081 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.035140 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.035190 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.035436 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.035475 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260619 20:27:15.035739 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.031131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:36657 (local address 127.0.0.1:35760)
0619 20:27:15.031949 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:15.031959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:15.032001 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:15.032221 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:15.032222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:15.032225 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:15.032260 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.032261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.033412 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.033413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.034370 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.034372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.035227 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.035227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.035296 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.035303 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 20:27:15.035349 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 20:27:15.035396 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":11,"thread_start_us":81,"threads_started":1}
I20260619 20:27:15.035866 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.031349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.21.151.2:33987)
0619 20:27:15.031526 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:15.031530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:15.031665 (+ 135us) server_negotiation.cc:407] Connection header received
0619 20:27:15.031719 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:15.031722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:15.031739 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:15.031817 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:15.032038 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.033154 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.034354 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.035070 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.035140 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.035190 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.035709 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.035752 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260619 20:27:15.036397 23637 tablet_service.cc:1467] Processing CreateTablet for tablet 6311104d04004339946f1a1d8f6e3cf6 (DEFAULT_TABLE table=test-table [id=2e0c664c86ee4b7480111baae2a6b725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:15.036466 23357 tablet_service.cc:1467] Processing CreateTablet for tablet 6311104d04004339946f1a1d8f6e3cf6 (DEFAULT_TABLE table=test-table [id=2e0c664c86ee4b7480111baae2a6b725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:15.036463 23497 tablet_service.cc:1467] Processing CreateTablet for tablet 6311104d04004339946f1a1d8f6e3cf6 (DEFAULT_TABLE table=test-table [id=2e0c664c86ee4b7480111baae2a6b725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:15.036723 23637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6311104d04004339946f1a1d8f6e3cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:15.036809 23497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6311104d04004339946f1a1d8f6e3cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:15.036809 23357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6311104d04004339946f1a1d8f6e3cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:15.039331 23726 tablet_bootstrap.cc:492] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: Bootstrap starting.
I20260619 20:27:15.040360 23726 tablet_bootstrap.cc:654] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:15.040622 23727 tablet_bootstrap.cc:492] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: Bootstrap starting.
I20260619 20:27:15.040719 23726 log.cc:824] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:15.041078 23728 tablet_bootstrap.cc:492] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: Bootstrap starting.
I20260619 20:27:15.041188 23727 tablet_bootstrap.cc:654] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:15.041424 23727 log.cc:824] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:15.041685 23728 tablet_bootstrap.cc:654] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:15.042013 23728 log.cc:824] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:15.042129 23727 tablet_bootstrap.cc:492] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: No bootstrap required, opened a new log
I20260619 20:27:15.042200 23727 ts_tablet_manager.cc:1397] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:15.042779 23726 tablet_bootstrap.cc:492] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: No bootstrap required, opened a new log
I20260619 20:27:15.042775 23728 tablet_bootstrap.cc:492] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: No bootstrap required, opened a new log
I20260619 20:27:15.042851 23726 ts_tablet_manager.cc:1397] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 20:27:15.042912 23728 ts_tablet_manager.cc:1397] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:27:15.044313 23727 raft_consensus.cc:348] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } }
I20260619 20:27:15.044492 23727 raft_consensus.cc:374] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:15.044528 23727 raft_consensus.cc:729] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ff040631cc45fba1fd8c457bfdfd8e, State: Initialized, Role: FOLLOWER
I20260619 20:27:15.044507 23728 raft_consensus.cc:348] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } }
I20260619 20:27:15.044660 23728 raft_consensus.cc:374] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:15.044690 23728 raft_consensus.cc:729] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c17f39faf346cd9bfca86919f7d5bf, State: Initialized, Role: FOLLOWER
I20260619 20:27:15.044683 23727 consensus_queue.cc:260] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } }
I20260619 20:27:15.044827 23728 consensus_queue.cc:260] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } }
I20260619 20:27:15.044967 23727 ts_tablet_manager.cc:1428] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:27:15.044915 23726 raft_consensus.cc:348] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } }
I20260619 20:27:15.045059 23726 raft_consensus.cc:374] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:15.045094 23563 heartbeater.cc:503] Master 127.21.151.62:38703 was elected leader, sending a full tablet report...
I20260619 20:27:15.045096 23728 ts_tablet_manager.cc:1428] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:27:15.045105 23726 raft_consensus.cc:729] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5113a8f802c448496dc15417afc74d6, State: Initialized, Role: FOLLOWER
I20260619 20:27:15.045243 23726 consensus_queue.cc:260] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } }
I20260619 20:27:15.045380 23423 heartbeater.cc:503] Master 127.21.151.62:38703 was elected leader, sending a full tablet report...
I20260619 20:27:15.045482 23726 ts_tablet_manager.cc:1428] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:27:15.045506 23703 heartbeater.cc:503] Master 127.21.151.62:38703 was elected leader, sending a full tablet report...
W20260619 20:27:15.074489 23564 tablet.cc:2369] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:27:15.168016 23424 tablet.cc:2369] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:27:15.240753 23704 tablet.cc:2369] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:27:15.274447 23732 raft_consensus.cc:482] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:15.274601 23732 raft_consensus.cc:504] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } }
I20260619 20:27:15.274999 23732 leader_election.cc:288] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c17f39faf346cd9bfca86919f7d5bf (127.21.151.1:40925), a5113a8f802c448496dc15417afc74d6 (127.21.151.3:36657)
I20260619 20:27:15.277757 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.275179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:40925 (local address 127.21.151.2:47645)
0619 20:27:15.275232 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:15.275245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:15.275293 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:15.275451 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:15.275452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:15.275453 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:15.275531 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.275533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.276197 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.276198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.276918 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.276920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.277489 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.277490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.277576 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.277579 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:15.277643 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 20:27:15.277682 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 20:27:15.277758 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.275266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:47645 (local address 127.21.151.1:40925)
0619 20:27:15.275326 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:15.275328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:15.275337 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:15.275364 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:15.275365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:15.275374 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:15.275448 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:15.275607 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.276101 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.277036 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.277388 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.277495 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.277503 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.277661 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.277698 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:27:15.277846 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.275306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:48783 (local address 127.21.151.3:36657)
0619 20:27:15.275363 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:15.275365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:15.275438 (+ 73us) server_negotiation.cc:407] Connection header received
0619 20:27:15.275501 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:15.275502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:15.275512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:15.275567 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:15.275771 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.276234 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.277146 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.277494 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.277578 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.277586 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.277745 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.277779 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:27:15.277859 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.275180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:36657 (local address 127.21.151.2:48783)
0619 20:27:15.275360 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:15.275371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:15.275437 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:15.275580 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:15.275580 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:15.275582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:15.275691 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.275692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.276322 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.276323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.277044 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.277046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.277578 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.277579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.277665 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.277668 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:15.277738 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:27:15.277779 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":12,"thread_start_us":90,"threads_started":1}
I20260619 20:27:15.278120 23377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6311104d04004339946f1a1d8f6e3cf6" candidate_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" is_pre_election: true
I20260619 20:27:15.278129 23657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6311104d04004339946f1a1d8f6e3cf6" candidate_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5113a8f802c448496dc15417afc74d6" is_pre_election: true
I20260619 20:27:15.278288 23657 raft_consensus.cc:2454] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ff040631cc45fba1fd8c457bfdfd8e in term 0.
I20260619 20:27:15.278290 23377 raft_consensus.cc:2454] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ff040631cc45fba1fd8c457bfdfd8e in term 0.
I20260619 20:27:15.278529 23451 leader_election.cc:302] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14ff040631cc45fba1fd8c457bfdfd8e, a5113a8f802c448496dc15417afc74d6, e5c17f39faf346cd9bfca86919f7d5bf; no voters:
I20260619 20:27:15.278664 23732 raft_consensus.cc:2781] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:15.278718 23732 raft_consensus.cc:482] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:15.278740 23732 raft_consensus.cc:3037] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:15.279709 23732 raft_consensus.cc:504] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } }
I20260619 20:27:15.279855 23732 leader_election.cc:288] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [CANDIDATE]: Term 1 election: Requested vote from peers e5c17f39faf346cd9bfca86919f7d5bf (127.21.151.1:40925), a5113a8f802c448496dc15417afc74d6 (127.21.151.3:36657)
I20260619 20:27:15.280014 23377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6311104d04004339946f1a1d8f6e3cf6" candidate_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c17f39faf346cd9bfca86919f7d5bf"
I20260619 20:27:15.280036 23657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6311104d04004339946f1a1d8f6e3cf6" candidate_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5113a8f802c448496dc15417afc74d6"
I20260619 20:27:15.280094 23377 raft_consensus.cc:3037] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:15.280118 23657 raft_consensus.cc:3037] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:15.280776 23377 raft_consensus.cc:2454] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ff040631cc45fba1fd8c457bfdfd8e in term 1.
I20260619 20:27:15.280792 23657 raft_consensus.cc:2454] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ff040631cc45fba1fd8c457bfdfd8e in term 1.
I20260619 20:27:15.280947 23451 leader_election.cc:302] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ff040631cc45fba1fd8c457bfdfd8e, a5113a8f802c448496dc15417afc74d6; no voters:
I20260619 20:27:15.281055 23732 raft_consensus.cc:2781] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:15.281210 23732 raft_consensus.cc:686] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 LEADER]: Becoming Leader. State: Replica: 14ff040631cc45fba1fd8c457bfdfd8e, State: Running, Role: LEADER
I20260619 20:27:15.281322 23732 consensus_queue.cc:237] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } }
I20260619 20:27:15.282168 23218 catalog_manager.cc:5495] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ff040631cc45fba1fd8c457bfdfd8e (127.21.151.2). New cstate: current_term: 1 leader_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } health_report { overall_health: HEALTHY } } }
I20260619 20:27:15.322222 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.319879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57752 (local address 127.21.151.2:33987)
0619 20:27:15.319944 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:15.319946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:15.319954 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:15.319991 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:15.319993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:15.320009 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:15.320065 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:15.320210 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.320702 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.321424 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.321605 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.321656 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.321879 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:15.321963 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:15.322035 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.322076 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.322093 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:27:15.324072 23657 raft_consensus.cc:1261] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6 [term 1 FOLLOWER]: Refusing update from remote peer 14ff040631cc45fba1fd8c457bfdfd8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:15.324080 23377 raft_consensus.cc:1261] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf [term 1 FOLLOWER]: Refusing update from remote peer 14ff040631cc45fba1fd8c457bfdfd8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:15.324440 23736 consensus_queue.cc:1035] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:15.324575 23732 consensus_queue.cc:1035] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:15.327885 23742 mvcc.cc:205] Tried to move back new op lower bound from 7298665821484449792 to 7298665821312806912. Current Snapshot: MvccSnapshot[applied={T|T < 7298665821484449792}]
I20260619 20:27:15.330842 23220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:15.330921 23220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:15.331719 23220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab: Sending DeleteTablet for 3 replicas of tablet 6311104d04004339946f1a1d8f6e3cf6
I20260619 20:27:15.332021 23357 tablet_service.cc:1514] Processing DeleteTablet for tablet 6311104d04004339946f1a1d8f6e3cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:15 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:35636
I20260619 20:27:15.332129 23637 tablet_service.cc:1514] Processing DeleteTablet for tablet 6311104d04004339946f1a1d8f6e3cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:15 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:35760
I20260619 20:27:15.332129 23497 tablet_service.cc:1514] Processing DeleteTablet for tablet 6311104d04004339946f1a1d8f6e3cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:15 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57740
I20260619 20:27:15.332334 23748 tablet_replica.cc:330] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: stopping tablet replica
I20260619 20:27:15.332515 23748 raft_consensus.cc:2229] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:15.332521 23750 tablet_replica.cc:330] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: stopping tablet replica
I20260619 20:27:15.332609 23750 raft_consensus.cc:2229] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:15.332542 23220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 20:27:15.332657 23748 raft_consensus.cc:2258] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 20:27:15.332682 23220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:15.332809 23750 raft_consensus.cc:2258] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:15.332973 23749 tablet_replica.cc:330] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: stopping tablet replica
I20260619 20:27:15.333138 23748 ts_tablet_manager.cc:1905] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:15.333166 23749 raft_consensus.cc:2229] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:15.333287 23749 raft_consensus.cc:2258] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:15.333741 23749 ts_tablet_manager.cc:1905] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:15.334213 23497 tablet_service.cc:1467] Processing CreateTablet for tablet da599d22d77d4f60a5f8ef4f037a0fe6 (DEFAULT_TABLE table=test-table [id=81e29fc32f444c39b99e4aaf110f89bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:15.334326 23497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da599d22d77d4f60a5f8ef4f037a0fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:15.335029 23748 ts_tablet_manager.cc:1918] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:15.335094 23748 log.cc:1196] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/6311104d04004339946f1a1d8f6e3cf6
I20260619 20:27:15.335331 23748 ts_tablet_manager.cc:1939] T 6311104d04004339946f1a1d8f6e3cf6 P e5c17f39faf346cd9bfca86919f7d5bf: Deleting consensus metadata
I20260619 20:27:15.335563 23749 ts_tablet_manager.cc:1918] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:15.335641 23749 log.cc:1196] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/6311104d04004339946f1a1d8f6e3cf6
I20260619 20:27:15.335705 23637 tablet_service.cc:1467] Processing CreateTablet for tablet da599d22d77d4f60a5f8ef4f037a0fe6 (DEFAULT_TABLE table=test-table [id=81e29fc32f444c39b99e4aaf110f89bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:15.335814 23637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da599d22d77d4f60a5f8ef4f037a0fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:15.335958 23727 tablet_bootstrap.cc:492] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e: Bootstrap starting.
I20260619 20:27:15.336020 23749 ts_tablet_manager.cc:1939] T 6311104d04004339946f1a1d8f6e3cf6 P 14ff040631cc45fba1fd8c457bfdfd8e: Deleting consensus metadata
I20260619 20:27:15.336360 23727 tablet_bootstrap.cc:654] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:15.336359 23205 catalog_manager.cc:4841] TS 14ff040631cc45fba1fd8c457bfdfd8e (127.21.151.2:33987): tablet 6311104d04004339946f1a1d8f6e3cf6 (table test-table [id=2e0c664c86ee4b7480111baae2a6b725]) successfully deleted
I20260619 20:27:15.336853 23206 catalog_manager.cc:4841] TS e5c17f39faf346cd9bfca86919f7d5bf (127.21.151.1:40925): tablet 6311104d04004339946f1a1d8f6e3cf6 (table test-table [id=2e0c664c86ee4b7480111baae2a6b725]) successfully deleted
I20260619 20:27:15.337126 23357 tablet_service.cc:1467] Processing CreateTablet for tablet da599d22d77d4f60a5f8ef4f037a0fe6 (DEFAULT_TABLE table=test-table [id=81e29fc32f444c39b99e4aaf110f89bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:15.337244 23357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da599d22d77d4f60a5f8ef4f037a0fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:15.338048 23728 tablet_bootstrap.cc:492] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf: Bootstrap starting.
I20260619 20:27:15.338265 23726 tablet_bootstrap.cc:492] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6: Bootstrap starting.
I20260619 20:27:15.338451 23728 tablet_bootstrap.cc:654] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:15.338693 23726 tablet_bootstrap.cc:654] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:15.338907 23727 tablet_bootstrap.cc:492] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e: No bootstrap required, opened a new log
I20260619 20:27:15.338948 23727 ts_tablet_manager.cc:1397] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:27:15.339073 23727 raft_consensus.cc:348] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.339159 23727 raft_consensus.cc:374] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:15.339179 23727 raft_consensus.cc:729] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ff040631cc45fba1fd8c457bfdfd8e, State: Initialized, Role: FOLLOWER
I20260619 20:27:15.339218 23727 consensus_queue.cc:260] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.339246 23726 tablet_bootstrap.cc:492] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6: No bootstrap required, opened a new log
I20260619 20:27:15.339282 23728 tablet_bootstrap.cc:492] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf: No bootstrap required, opened a new log
I20260619 20:27:15.339288 23726 ts_tablet_manager.cc:1397] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:15.339298 23727 ts_tablet_manager.cc:1428] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:15.339318 23728 ts_tablet_manager.cc:1397] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:15.339438 23726 raft_consensus.cc:348] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.339454 23728 raft_consensus.cc:348] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.339524 23728 raft_consensus.cc:374] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:15.339524 23726 raft_consensus.cc:374] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:15.339547 23728 raft_consensus.cc:729] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c17f39faf346cd9bfca86919f7d5bf, State: Initialized, Role: FOLLOWER
I20260619 20:27:15.339547 23726 raft_consensus.cc:729] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5113a8f802c448496dc15417afc74d6, State: Initialized, Role: FOLLOWER
I20260619 20:27:15.339586 23726 consensus_queue.cc:260] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.339584 23728 consensus_queue.cc:260] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.339687 23726 ts_tablet_manager.cc:1428] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:15.339706 23728 ts_tablet_manager.cc:1428] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:15.340504 23750 ts_tablet_manager.cc:1905] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:15.341985 23750 ts_tablet_manager.cc:1918] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:15.342056 23750 log.cc:1196] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/6311104d04004339946f1a1d8f6e3cf6
I20260619 20:27:15.342314 23750 ts_tablet_manager.cc:1939] T 6311104d04004339946f1a1d8f6e3cf6 P a5113a8f802c448496dc15417afc74d6: Deleting consensus metadata
I20260619 20:27:15.342729 23207 catalog_manager.cc:4841] TS a5113a8f802c448496dc15417afc74d6 (127.21.151.3:36657): tablet 6311104d04004339946f1a1d8f6e3cf6 (table test-table [id=2e0c664c86ee4b7480111baae2a6b725]) successfully deleted
I20260619 20:27:15.733044 23733 raft_consensus.cc:482] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:15.733162 23733 raft_consensus.cc:504] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.733506 23733 leader_election.cc:288] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ff040631cc45fba1fd8c457bfdfd8e (127.21.151.2:33987), a5113a8f802c448496dc15417afc74d6 (127.21.151.3:36657)
I20260619 20:27:15.736716 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.733794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:50113 (local address 127.21.151.3:36657)
0619 20:27:15.733868 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:15.733870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:15.733994 (+ 124us) server_negotiation.cc:407] Connection header received
0619 20:27:15.734069 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:15.734070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:15.734081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:15.734158 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:15.734323 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.734804 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.735907 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.736266 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.736353 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.736362 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.736506 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.736543 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:27:15.736719 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.733733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:36657 (local address 127.21.151.1:50113)
0619 20:27:15.733917 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:15.733926 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:15.734009 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:15.734163 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:15.734164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:15.734165 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:15.734246 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.734248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.734892 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.734893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.735819 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.735821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.736352 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.736353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.736424 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.736436 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 20:27:15.736503 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 20:27:15.736542 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":37,"thread_start_us":105,"threads_started":1}
I20260619 20:27:15.736832 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.733800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:43599 (local address 127.21.151.2:33987)
0619 20:27:15.733853 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:15.733855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:15.733991 (+ 136us) server_negotiation.cc:407] Connection header received
0619 20:27:15.734085 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:15.734086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:15.734106 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:15.734168 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:15.734345 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.734813 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.735927 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.736289 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.736379 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.736388 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.736552 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.736595 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:27:15.736846 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.733734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:33987 (local address 127.21.151.1:43599)
0619 20:27:15.733900 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:15.733912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:15.734009 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:15.734187 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:15.734188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:15.734190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:15.734271 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.734273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.734946 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.734948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.735822 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.735824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.736379 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.736380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.736464 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.736467 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:15.736546 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:27:15.736683 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260619 20:27:15.737242 23657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da599d22d77d4f60a5f8ef4f037a0fe6" candidate_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5113a8f802c448496dc15417afc74d6" is_pre_election: true
I20260619 20:27:15.737246 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da599d22d77d4f60a5f8ef4f037a0fe6" candidate_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" is_pre_election: true
I20260619 20:27:15.737349 23657 raft_consensus.cc:2454] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c17f39faf346cd9bfca86919f7d5bf in term 0.
I20260619 20:27:15.737402 23517 raft_consensus.cc:2454] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c17f39faf346cd9bfca86919f7d5bf in term 0.
I20260619 20:27:15.737550 23311 leader_election.cc:302] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5113a8f802c448496dc15417afc74d6, e5c17f39faf346cd9bfca86919f7d5bf; no voters:
I20260619 20:27:15.737733 23733 raft_consensus.cc:2781] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:15.737794 23733 raft_consensus.cc:482] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:15.737813 23733 raft_consensus.cc:3037] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:15.738622 23733 raft_consensus.cc:504] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.738786 23733 leader_election.cc:288] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [CANDIDATE]: Term 1 election: Requested vote from peers 14ff040631cc45fba1fd8c457bfdfd8e (127.21.151.2:33987), a5113a8f802c448496dc15417afc74d6 (127.21.151.3:36657)
I20260619 20:27:15.738955 23657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da599d22d77d4f60a5f8ef4f037a0fe6" candidate_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5113a8f802c448496dc15417afc74d6"
I20260619 20:27:15.739060 23657 raft_consensus.cc:3037] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:15.739751 23657 raft_consensus.cc:2454] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c17f39faf346cd9bfca86919f7d5bf in term 1.
I20260619 20:27:15.739874 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da599d22d77d4f60a5f8ef4f037a0fe6" candidate_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ff040631cc45fba1fd8c457bfdfd8e"
I20260619 20:27:15.739898 23311 leader_election.cc:302] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5113a8f802c448496dc15417afc74d6, e5c17f39faf346cd9bfca86919f7d5bf; no voters:
I20260619 20:27:15.739957 23517 raft_consensus.cc:3037] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:15.740021 23733 raft_consensus.cc:2781] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:15.740208 23733 raft_consensus.cc:686] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 1 LEADER]: Becoming Leader. State: Replica: e5c17f39faf346cd9bfca86919f7d5bf, State: Running, Role: LEADER
I20260619 20:27:15.740283 23733 consensus_queue.cc:237] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.740934 23517 raft_consensus.cc:2454] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c17f39faf346cd9bfca86919f7d5bf in term 1.
I20260619 20:27:15.741106 23218 catalog_manager.cc:5495] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to e5c17f39faf346cd9bfca86919f7d5bf (127.21.151.1). New cstate: current_term: 1 leader_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } health_report { overall_health: UNKNOWN } } }
Jun 19 20:27:15 dist-test-slave-f9rk krb5kdc[23175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900834, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:15.810835 23764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.804403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.21.151.62:38703)
0619 20:27:15.804566 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:15.804568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:15.804598 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:27:15.804670 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:15.804671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:15.804694 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:15.804776 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:15.805220 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.805720 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.806411 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.806612 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.806666 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.808631 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:15.808633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:15.808633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:15.808639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:15.809867 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:15.810212 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:15.810213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:15.810213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:15.810265 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:15.810427 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:15.810427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:15.810428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:15.810534 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:15.810583 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:15.810697 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 20:27:15.810765 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260619 20:27:15.810992 23763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:15.804346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38703 (local address 127.0.0.1:53308)
0619 20:27:15.804524 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:15.804537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:15.804616 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:15.804798 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:15.804799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:15.805105 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:15.805157 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.805159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.805784 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.805785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.806339 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:15.806341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:15.806677 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:15.806677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:15.806717 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:15.807188 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:15.807192 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:15.808553 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:15.809937 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:15.809943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:15.809944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:15.810149 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:15.810321 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:15.810322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:15.810322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:15.810377 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:15.810586 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:15.810592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:15.810631 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 20:27:15.810683 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:27:15.810832 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":62,"thread_start_us":87,"threads_started":1}
I20260619 20:27:15.812862 23218 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:53308:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 20:27:15.813071 23218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:15.814450 23497 tablet_service.cc:1467] Processing CreateTablet for tablet e6601b5a6eb74a6fb47c7801a363792f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65d325f5c0c04c77843e8ea6f914c83a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:27:15.814457 23357 tablet_service.cc:1467] Processing CreateTablet for tablet e6601b5a6eb74a6fb47c7801a363792f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65d325f5c0c04c77843e8ea6f914c83a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:27:15.814488 23637 tablet_service.cc:1467] Processing CreateTablet for tablet e6601b5a6eb74a6fb47c7801a363792f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65d325f5c0c04c77843e8ea6f914c83a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:27:15.814610 23497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6601b5a6eb74a6fb47c7801a363792f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:15.814616 23357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6601b5a6eb74a6fb47c7801a363792f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:15.814642 23637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6601b5a6eb74a6fb47c7801a363792f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:15.815713 23726 tablet_bootstrap.cc:492] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: Bootstrap starting.
I20260619 20:27:15.816056 23728 tablet_bootstrap.cc:492] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: Bootstrap starting.
I20260619 20:27:15.816071 23727 tablet_bootstrap.cc:492] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: Bootstrap starting.
I20260619 20:27:15.816147 23726 tablet_bootstrap.cc:654] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:15.816388 23727 tablet_bootstrap.cc:654] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:15.816483 23728 tablet_bootstrap.cc:654] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:15.816885 23726 tablet_bootstrap.cc:492] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: No bootstrap required, opened a new log
I20260619 20:27:15.816931 23726 ts_tablet_manager.cc:1397] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:15.817137 23727 tablet_bootstrap.cc:492] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: No bootstrap required, opened a new log
I20260619 20:27:15.817101 23726 raft_consensus.cc:348] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.817178 23726 raft_consensus.cc:374] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:15.817189 23727 ts_tablet_manager.cc:1397] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:15.817201 23726 raft_consensus.cc:729] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5113a8f802c448496dc15417afc74d6, State: Initialized, Role: FOLLOWER
I20260619 20:27:15.817250 23726 consensus_queue.cc:260] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.817281 23728 tablet_bootstrap.cc:492] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: No bootstrap required, opened a new log
I20260619 20:27:15.817324 23728 ts_tablet_manager.cc:1397] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:15.817333 23726 ts_tablet_manager.cc:1428] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:15.817296 23727 raft_consensus.cc:348] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.817377 23727 raft_consensus.cc:374] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:15.817409 23727 raft_consensus.cc:729] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ff040631cc45fba1fd8c457bfdfd8e, State: Initialized, Role: FOLLOWER
I20260619 20:27:15.817449 23727 consensus_queue.cc:260] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.817497 23728 raft_consensus.cc:348] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.817538 23727 ts_tablet_manager.cc:1428] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:15.817575 23728 raft_consensus.cc:374] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:15.817598 23728 raft_consensus.cc:729] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c17f39faf346cd9bfca86919f7d5bf, State: Initialized, Role: FOLLOWER
I20260619 20:27:15.817642 23728 consensus_queue.cc:260] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:15.817687 23736 tablet_replica.cc:439] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } }
I20260619 20:27:15.817728 23728 ts_tablet_manager.cc:1428] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:15.817766 23736 tablet_replica.cc:442] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:15.817816 23733 tablet_replica.cc:439] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } }
I20260619 20:27:15.817888 23733 tablet_replica.cc:442] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:15.817862 23734 tablet_replica.cc:439] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } }
I20260619 20:27:15.817934 23734 tablet_replica.cc:442] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:16.019109 23734 raft_consensus.cc:482] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:16.019205 23734 raft_consensus.cc:504] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:16.019522 23734 leader_election.cc:288] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c17f39faf346cd9bfca86919f7d5bf (127.21.151.1:40925), 14ff040631cc45fba1fd8c457bfdfd8e (127.21.151.2:33987)
I20260619 20:27:16.023599 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.019817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:58171 (local address 127.21.151.2:33987)
0619 20:27:16.019858 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:16.019860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:16.019993 (+ 133us) server_negotiation.cc:407] Connection header received
0619 20:27:16.020056 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:16.020058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:16.020068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:16.020136 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:16.020314 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.021082 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.022646 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.023203 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.023281 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.023289 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:16.023462 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 20:27:16.023511 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:27:16.023633 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.019736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:33987 (local address 127.21.151.3:58171)
0619 20:27:16.019893 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:16.019906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:16.019992 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:16.020159 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:16.020160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:16.020164 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:16.020237 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:16.020240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.021183 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.021183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:16.022548 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:16.022551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.023295 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.023296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:16.023379 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.023384 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:27:16.023446 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:27:16.023505 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":83,"thread_start_us":68,"threads_started":1}
I20260619 20:27:16.023849 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.019738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:40925 (local address 127.21.151.3:48579)
0619 20:27:16.019904 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:16.019914 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:16.019992 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:16.020159 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:16.020160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:16.020162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:16.020237 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:16.020239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.021166 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.021166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:16.022548 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:16.022551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.023296 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.023297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:16.023385 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.023389 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:16.023446 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:27:16.023505 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":105,"thread_start_us":61,"threads_started":1}
I20260619 20:27:16.024049 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.019798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:48579 (local address 127.21.151.1:40925)
0619 20:27:16.019917 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:16.019918 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:16.019978 (+ 60us) server_negotiation.cc:407] Connection header received
0619 20:27:16.020062 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:16.020063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:16.020074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:16.020139 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:16.020314 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.021082 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.022645 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.023203 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.023282 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.023289 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:16.023555 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 20:27:16.023607 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260619 20:27:16.024223 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6601b5a6eb74a6fb47c7801a363792f" candidate_uuid: "a5113a8f802c448496dc15417afc74d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" is_pre_election: true
I20260619 20:27:16.024255 23377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6601b5a6eb74a6fb47c7801a363792f" candidate_uuid: "a5113a8f802c448496dc15417afc74d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" is_pre_election: true
I20260619 20:27:16.024350 23517 raft_consensus.cc:2454] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5113a8f802c448496dc15417afc74d6 in term 0.
I20260619 20:27:16.024367 23377 raft_consensus.cc:2454] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5113a8f802c448496dc15417afc74d6 in term 0.
I20260619 20:27:16.024549 23589 leader_election.cc:302] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ff040631cc45fba1fd8c457bfdfd8e, a5113a8f802c448496dc15417afc74d6; no voters:
I20260619 20:27:16.024677 23734 raft_consensus.cc:2781] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:16.024724 23734 raft_consensus.cc:482] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:16.024755 23734 raft_consensus.cc:3037] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:16.025419 23734 raft_consensus.cc:504] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:16.025585 23734 leader_election.cc:288] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [CANDIDATE]: Term 1 election: Requested vote from peers e5c17f39faf346cd9bfca86919f7d5bf (127.21.151.1:40925), 14ff040631cc45fba1fd8c457bfdfd8e (127.21.151.2:33987)
I20260619 20:27:16.025702 23377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6601b5a6eb74a6fb47c7801a363792f" candidate_uuid: "a5113a8f802c448496dc15417afc74d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c17f39faf346cd9bfca86919f7d5bf"
I20260619 20:27:16.025771 23377 raft_consensus.cc:3037] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:16.025749 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6601b5a6eb74a6fb47c7801a363792f" candidate_uuid: "a5113a8f802c448496dc15417afc74d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ff040631cc45fba1fd8c457bfdfd8e"
I20260619 20:27:16.025843 23517 raft_consensus.cc:3037] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:16.026331 23377 raft_consensus.cc:2454] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5113a8f802c448496dc15417afc74d6 in term 1.
I20260619 20:27:16.026499 23517 raft_consensus.cc:2454] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5113a8f802c448496dc15417afc74d6 in term 1.
I20260619 20:27:16.026497 23590 leader_election.cc:302] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5113a8f802c448496dc15417afc74d6, e5c17f39faf346cd9bfca86919f7d5bf; no voters:
I20260619 20:27:16.026669 23734 raft_consensus.cc:2781] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:16.026806 23734 raft_consensus.cc:686] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [term 1 LEADER]: Becoming Leader. State: Replica: a5113a8f802c448496dc15417afc74d6, State: Running, Role: LEADER
I20260619 20:27:16.026871 23734 consensus_queue.cc:237] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } }
I20260619 20:27:16.027263 23771 tablet_replica.cc:439] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: TxnStatusTablet state changed. Reason: New leader a5113a8f802c448496dc15417afc74d6. Latest consensus state: current_term: 1 leader_uuid: "a5113a8f802c448496dc15417afc74d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } }
I20260619 20:27:16.027390 23771 tablet_replica.cc:442] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:27:16.027573 23773 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 20:27:16.027719 23773 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 20:27:16.027717 23218 catalog_manager.cc:5495] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5113a8f802c448496dc15417afc74d6 (127.21.151.3). New cstate: current_term: 1 leader_uuid: "a5113a8f802c448496dc15417afc74d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } health_report { overall_health: HEALTHY } } }
I20260619 20:27:16.046085 23763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.043086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:36657 (local address 127.0.0.1:35774)
0619 20:27:16.043145 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:16.043155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:16.043196 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:16.043509 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:16.043510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:16.043512 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:16.043556 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:16.043558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.044505 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.044506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:16.045134 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:16.045136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.045608 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.045609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:16.045654 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.045667 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:16.045951 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:16.045958 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 20:27:16.045992 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 20:27:16.046029 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 20:27:16.046113 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.043284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35774 (local address 127.21.151.3:36657)
0619 20:27:16.043350 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:16.043352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:16.043362 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:16.043400 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:16.043401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:16.043419 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:16.043494 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:16.043663 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.044416 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.045218 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.045525 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.045604 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.045757 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:16.045880 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:16.045960 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:16.045997 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 20:27:16.046016 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:27:16.047323 23377 raft_consensus.cc:1261] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf [term 1 FOLLOWER]: Refusing update from remote peer a5113a8f802c448496dc15417afc74d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:16.047585 23517 raft_consensus.cc:1261] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 FOLLOWER]: Refusing update from remote peer a5113a8f802c448496dc15417afc74d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:16.047657 23771 consensus_queue.cc:1035] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:16.047842 23771 consensus_queue.cc:1035] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:16.048293 23733 tablet_replica.cc:439] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: TxnStatusTablet state changed. Reason: New leader a5113a8f802c448496dc15417afc74d6. Latest consensus state: current_term: 1 leader_uuid: "a5113a8f802c448496dc15417afc74d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } }
I20260619 20:27:16.048381 23733 tablet_replica.cc:442] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:16.048660 23736 tablet_replica.cc:439] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: TxnStatusTablet state changed. Reason: New leader a5113a8f802c448496dc15417afc74d6. Latest consensus state: current_term: 1 leader_uuid: "a5113a8f802c448496dc15417afc74d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } }
I20260619 20:27:16.048749 23736 tablet_replica.cc:442] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:16.049481 23734 tablet_replica.cc:439] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5113a8f802c448496dc15417afc74d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } }
I20260619 20:27:16.049561 23734 tablet_replica.cc:442] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:27:16.049541 23733 tablet_replica.cc:439] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5113a8f802c448496dc15417afc74d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } }
I20260619 20:27:16.049623 23733 tablet_replica.cc:442] T e6601b5a6eb74a6fb47c7801a363792f P e5c17f39faf346cd9bfca86919f7d5bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:16.050295 23736 tablet_replica.cc:439] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5113a8f802c448496dc15417afc74d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } }
I20260619 20:27:16.050379 23736 tablet_replica.cc:442] T e6601b5a6eb74a6fb47c7801a363792f P 14ff040631cc45fba1fd8c457bfdfd8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:16.050072 23783 tablet_replica.cc:439] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5113a8f802c448496dc15417afc74d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c17f39faf346cd9bfca86919f7d5bf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 40925 } } peers { permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 } } peers { permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 } } }
I20260619 20:27:16.050860 23783 tablet_replica.cc:442] T e6601b5a6eb74a6fb47c7801a363792f P a5113a8f802c448496dc15417afc74d6: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:27:16.054262 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.051596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.21.151.1:40925)
0619 20:27:16.051655 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:16.051657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:16.051693 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:27:16.051779 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:16.051780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:16.051796 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:16.051846 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:16.051968 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.052411 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.053539 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.053723 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.053778 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.053964 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:16.054056 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:16.054104 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:16.054173 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 20:27:16.054189 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:27:16.059681 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.056840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:35405 (local address 127.21.151.3:36657)
0619 20:27:16.056891 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:16.056893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:16.057005 (+ 112us) server_negotiation.cc:407] Connection header received
0619 20:27:16.057059 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:16.057060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:16.057069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:16.057132 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:16.057246 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.057969 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.058906 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.059208 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.059271 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.059440 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:16.059471 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:16.059528 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:16.059573 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 20:27:16.059588 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:27:16.059780 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.056760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:36657 (local address 127.21.151.1:35405)
0619 20:27:16.056927 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:16.056939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:16.057012 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:16.057138 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:16.057139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:16.057140 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:16.057190 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:16.057191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.058047 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.058048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:16.058808 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:16.058810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.059282 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.059283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:16.059338 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.059353 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:16.059530 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:16.059531 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:27:16.059570 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 20:27:16.059605 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260619 20:27:16.062007 23657 raft_consensus.cc:1261] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6 [term 1 FOLLOWER]: Refusing update from remote peer e5c17f39faf346cd9bfca86919f7d5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:16.062041 23517 raft_consensus.cc:1261] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 FOLLOWER]: Refusing update from remote peer e5c17f39faf346cd9bfca86919f7d5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:16.062314 23733 consensus_queue.cc:1035] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ff040631cc45fba1fd8c457bfdfd8e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:16.062459 23756 consensus_queue.cc:1035] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5113a8f802c448496dc15417afc74d6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:16.063776 23777 mvcc.cc:205] Tried to move back new op lower bound from 7298665824508235776 to 7298665823192846336. Current Snapshot: MvccSnapshot[applied={T|T < 7298665824508235776}]
I20260619 20:27:16.068706 23779 mvcc.cc:205] Tried to move back new op lower bound from 7298665824508235776 to 7298665823192846336. Current Snapshot: MvccSnapshot[applied={T|T < 7298665824508235776}]
I20260619 20:27:16.072954 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.068593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:34483 (local address 127.21.151.1:40925)
0619 20:27:16.068642 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:16.068645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:16.069767 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 20:27:16.069844 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:16.069846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:16.069858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:16.069927 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:16.070102 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.070736 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.072099 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.072332 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.072437 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.072649 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:16.072686 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:16.072764 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:16.072848 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 20:27:16.072872 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:27:16.073004 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.068447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:40925 (local address 127.21.151.3:34483)
0619 20:27:16.069685 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:16.069697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:16.069781 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:16.069931 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:16.069931 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:16.069933 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:16.070023 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:16.070024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.070841 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.070842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:16.071987 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:16.071990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.072436 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.072437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:16.072526 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.072541 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:16.072777 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:16.072780 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:16.072843 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 20:27:16.072895 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"thread_start_us":82,"threads_started":1}
I20260619 20:27:16.081672 23219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:16.081792 23219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:16.082468 23219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26cd84fdae4f463391a9d330567c8dab: Sending DeleteTablet for 3 replicas of tablet da599d22d77d4f60a5f8ef4f037a0fe6
I20260619 20:27:16.082707 23497 tablet_service.cc:1514] Processing DeleteTablet for tablet da599d22d77d4f60a5f8ef4f037a0fe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:16 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57740
I20260619 20:27:16.082742 23637 tablet_service.cc:1514] Processing DeleteTablet for tablet da599d22d77d4f60a5f8ef4f037a0fe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:16 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:35760
I20260619 20:27:16.082911 23357 tablet_service.cc:1514] Processing DeleteTablet for tablet da599d22d77d4f60a5f8ef4f037a0fe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:16 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:35636
I20260619 20:27:16.083077 23797 tablet_replica.cc:330] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e: stopping tablet replica
I20260619 20:27:16.083134 23797 raft_consensus.cc:2229] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:16.083173 23797 raft_consensus.cc:2258] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:16.083421 23797 ts_tablet_manager.cc:1905] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:16.083709 23799 tablet_replica.cc:330] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf: stopping tablet replica
I20260619 20:27:16.083827 23799 raft_consensus.cc:2229] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:16.083950 23799 raft_consensus.cc:2258] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:16.084254 23799 ts_tablet_manager.cc:1905] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:16.085180 23798 tablet_replica.cc:330] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6: stopping tablet replica
I20260619 20:27:16.085270 23798 raft_consensus.cc:2229] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:16.085314 23798 raft_consensus.cc:2258] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:16.085435 23797 ts_tablet_manager.cc:1918] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:16.085502 23797 log.cc:1196] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/da599d22d77d4f60a5f8ef4f037a0fe6
I20260619 20:27:16.085757 23797 ts_tablet_manager.cc:1939] T da599d22d77d4f60a5f8ef4f037a0fe6 P 14ff040631cc45fba1fd8c457bfdfd8e: Deleting consensus metadata
I20260619 20:27:16.086045 23205 catalog_manager.cc:4841] TS 14ff040631cc45fba1fd8c457bfdfd8e (127.21.151.2:33987): tablet da599d22d77d4f60a5f8ef4f037a0fe6 (table test-table [id=81e29fc32f444c39b99e4aaf110f89bb]) successfully deleted
I20260619 20:27:16.086221 23799 ts_tablet_manager.cc:1918] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:16.086319 23799 log.cc:1196] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/da599d22d77d4f60a5f8ef4f037a0fe6
I20260619 20:27:16.086329 23798 ts_tablet_manager.cc:1905] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:16.086609 23799 ts_tablet_manager.cc:1939] T da599d22d77d4f60a5f8ef4f037a0fe6 P e5c17f39faf346cd9bfca86919f7d5bf: Deleting consensus metadata
I20260619 20:27:16.086908 23206 catalog_manager.cc:4841] TS e5c17f39faf346cd9bfca86919f7d5bf (127.21.151.1:40925): tablet da599d22d77d4f60a5f8ef4f037a0fe6 (table test-table [id=81e29fc32f444c39b99e4aaf110f89bb]) successfully deleted
I20260619 20:27:16.088390 23798 ts_tablet_manager.cc:1918] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:16.088449 23798 log.cc:1196] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/da599d22d77d4f60a5f8ef4f037a0fe6
I20260619 20:27:16.088693 23798 ts_tablet_manager.cc:1939] T da599d22d77d4f60a5f8ef4f037a0fe6 P a5113a8f802c448496dc15417afc74d6: Deleting consensus metadata
I20260619 20:27:16.088968 23207 catalog_manager.cc:4841] TS a5113a8f802c448496dc15417afc74d6 (127.21.151.3:36657): tablet da599d22d77d4f60a5f8ef4f037a0fe6 (table test-table [id=81e29fc32f444c39b99e4aaf110f89bb]) successfully deleted
Jun 19 20:27:16 dist-test-slave-f9rk krb5kdc[23175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900835, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.151.1@KRBTEST.COM
I20260619 20:27:16.099396 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.089179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.21.151.1:40925)
0619 20:27:16.089223 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:16.089225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:16.089452 (+ 227us) server_negotiation.cc:407] Connection header received
0619 20:27:16.089532 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:16.089532 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:16.089542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:16.089593 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:16.090239 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.090990 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.092076 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.092377 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.092451 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.096026 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:16.096034 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:16.096034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:16.096042 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:16.098102 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:16.098484 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:16.098487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:16.098487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:16.098563 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:16.098806 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:16.098806 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:16.098807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:16.098962 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:16.099109 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:16.099192 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 20:27:16.099273 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:27:16.099754 23417 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35664
I20260619 20:27:16.108403 23764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:16.100719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.21.151.62:38703)
0619 20:27:16.100767 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:16.100769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:16.100977 (+ 208us) server_negotiation.cc:407] Connection header received
0619 20:27:16.101026 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:16.101028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:16.101041 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:16.101090 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:16.101727 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.102499 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.103537 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:16.103886 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:16.103957 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:16.105353 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:16.105355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:16.105355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:16.105363 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:16.107058 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:16.107443 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:16.107444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:16.107445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:16.107507 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:16.107804 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:16.107805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:16.107805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:16.107947 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:16.108061 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:16.108204 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 20:27:16.108295 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 20:27:16.108711 23219 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53324
I20260619 20:27:16.109308 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 23292
I20260619 20:27:16.116082 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 23432
I20260619 20:27:16.121997 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 23572
I20260619 20:27:16.127010 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 23188
2026-06-19T20:27:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2947 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:16 dist-test-slave-f9rk krb5kdc[23806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:16 dist-test-slave-f9rk krb5kdc[23806](info): set up 2 sockets
Jun 19 20:27:16 dist-test-slave-f9rk krb5kdc[23806](info): commencing operation
krb5kdc: starting...
W20260619 20:27:17.216394 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:17.330888 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:35593
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:35593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:17.412098 23819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:17.412356 23819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:17.412391 23819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:17.413488 23819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:17.413558 23819 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 20:27:17.413580 23819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:17.413601 23819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:17.413630 23819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:17.413653 23819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:17.415057 23819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:35593
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:35593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:17.415325 23819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:17.415541 23819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:17.420358 23827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:17.420358 23825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:17.420574 23819 server_base.cc:1034] running on GCE node
W20260619 20:27:17.420375 23824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:17.420908 23819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:17.421195 23819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:17.422348 23819 hybrid_clock.cc:648] HybridClock initialized: now 1781900837422332 us; error 52 us; skew 500 ppm
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:17.424885 23819 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:17.425393 23819 webserver.cc:460] Webserver started at http://127.21.151.62:39827/ using document root <none> and password file <none>
I20260619 20:27:17.425623 23819 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:17.425693 23819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:17.425837 23819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:17.426913 23819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "2b7a22643b794a859b0ae0ab15ef8d94"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "16b4b8e66dc64ca22f58315f3162492f"
server_key_iv: "71891452a69251cc3f3f824c6ddf6250"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.427285 23819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "2b7a22643b794a859b0ae0ab15ef8d94"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "16b4b8e66dc64ca22f58315f3162492f"
server_key_iv: "71891452a69251cc3f3f824c6ddf6250"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.428663 23819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:17.429400 23834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:17.429575 23819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:17.429677 23819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "2b7a22643b794a859b0ae0ab15ef8d94"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "16b4b8e66dc64ca22f58315f3162492f"
server_key_iv: "71891452a69251cc3f3f824c6ddf6250"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.429759 23819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:17.461488 23819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:17.463402 23819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:17.463569 23819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:17.468379 23819 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:35593
I20260619 20:27:17.468441 23896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:35593 every 8 connection(s)
I20260619 20:27:17.468808 23819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:17.469444 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:17.471674 23897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94: Bootstrap starting.
I20260619 20:27:17.472213 23897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:17.472435 23897 log.cc:824] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:17.473047 23897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94: No bootstrap required, opened a new log
I20260619 20:27:17.475054 23897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7a22643b794a859b0ae0ab15ef8d94" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35593 } }
I20260619 20:27:17.475239 23897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:17.475274 23897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7a22643b794a859b0ae0ab15ef8d94, State: Initialized, Role: FOLLOWER
I20260619 20:27:17.475411 23897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7a22643b794a859b0ae0ab15ef8d94" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35593 } }
I20260619 20:27:17.475489 23897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:17.475517 23897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:17.475545 23897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:17.475888 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 23819
I20260619 20:27:17.475973 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:17.476143 22108 external_mini_cluster.cc:1428] Setting key 3c9e92cc47ec668805721b751b486305
I20260619 20:27:17.476326 23897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7a22643b794a859b0ae0ab15ef8d94" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35593 } }
I20260619 20:27:17.476428 23897 leader_election.cc:302] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b7a22643b794a859b0ae0ab15ef8d94; no voters:
I20260619 20:27:17.476603 23897 leader_election.cc:288] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:17.476770 23902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:17.476850 23897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:17.476946 23902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [term 1 LEADER]: Becoming Leader. State: Replica: 2b7a22643b794a859b0ae0ab15ef8d94, State: Running, Role: LEADER
I20260619 20:27:17.477028 23902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7a22643b794a859b0ae0ab15ef8d94" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35593 } }
I20260619 20:27:17.477367 23902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7a22643b794a859b0ae0ab15ef8d94. Latest consensus state: current_term: 1 leader_uuid: "2b7a22643b794a859b0ae0ab15ef8d94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7a22643b794a859b0ae0ab15ef8d94" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35593 } } }
I20260619 20:27:17.477440 23902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:17.477583 23900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b7a22643b794a859b0ae0ab15ef8d94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7a22643b794a859b0ae0ab15ef8d94" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35593 } } }
I20260619 20:27:17.477648 23900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:17.477777 23910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:17.478317 23910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:17.480857 23910 catalog_manager.cc:1347] Generated new cluster ID: 7f33a2ce90e949c6a629bc581b981c2c
I20260619 20:27:17.480909 23910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:17.486090 23910 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:17.486848 23910 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:17.488885 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.476658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.21.151.62:35593)
0619 20:27:17.478274 (+ 1616us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.478284 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.478320 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:27:17.478438 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.478445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.478509 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.479389 (+ 880us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:17.479486 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.480451 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.481558 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.482160 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.482256 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.485167 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:17.485172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:17.485172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:17.485185 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:17.487244 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.487707 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.487710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.487710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.487809 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.488022 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.488023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.488023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.488289 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:17.488426 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.488544 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.488680 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1553,"thread_start_us":57,"threads_started":1}
I20260619 20:27:17.492322 23910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94: Generated new TSK 0
I20260619 20:27:17.492506 23910 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:27:17.518030 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:35593
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:17.598188 23923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:17.598379 23923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:17.598403 23923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:17.599534 23923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:17.599576 23923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:17.599596 23923 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 20:27:17.599610 23923 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:27:17.599684 23923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:17.601130 23923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.151.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:17.601363 23923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:17.601600 23923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:27:17.606652 23923 server_base.cc:1034] running on GCE node
W20260619 20:27:17.606648 23929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:17.606648 23928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:17.606767 23931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:17.607096 23923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:17.607287 23923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:17.608426 23923 hybrid_clock.cc:648] HybridClock initialized: now 1781900837608416 us; error 26 us; skew 500 ppm
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:17.610883 23923 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:27:17.611372 23923 webserver.cc:460] Webserver started at http://127.21.151.1:42211/ using document root <none> and password file <none>
I20260619 20:27:17.611588 23923 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:17.611687 23923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:17.611837 23923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:17.612900 23923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "55f4a0f200a6437a9a734e071b535003"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "b1db02f4752580f05fe4d188b1808a61"
server_key_iv: "2c54f84fb781e5d86b373862c835ebb1"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.613255 23923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "55f4a0f200a6437a9a734e071b535003"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "b1db02f4752580f05fe4d188b1808a61"
server_key_iv: "2c54f84fb781e5d86b373862c835ebb1"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.614485 23923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:17.615162 23938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:17.615317 23923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:17.615412 23923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "55f4a0f200a6437a9a734e071b535003"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "b1db02f4752580f05fe4d188b1808a61"
server_key_iv: "2c54f84fb781e5d86b373862c835ebb1"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.615491 23923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:17.627251 23923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:17.629040 23923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:17.629194 23923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:17.629719 23923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:17.629776 23923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:17.629812 23923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:17.629833 23923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:17.637090 23923 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:41263
I20260619 20:27:17.637159 24053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:41263 every 8 connection(s)
I20260619 20:27:17.637452 23923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:27:17.639628 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.630125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:38703 (local address 127.21.151.62:35593)
0619 20:27:17.630199 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.630201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.630380 (+ 179us) server_negotiation.cc:407] Connection header received
0619 20:27:17.630838 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.630840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.630853 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.630931 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:17.631566 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.632408 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.633736 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.634106 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.634189 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.636422 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:17.636424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:17.636424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:17.636433 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:17.638197 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.638602 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.638604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.638604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.638667 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.638938 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.638938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.638939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.639063 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:17.639141 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.639413 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.639511 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:27:17.639782 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.630102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:35593 (local address 127.21.151.1:38703)
0619 20:27:17.630250 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:17.630287 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:17.630748 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:17.630994 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:17.630997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:17.631230 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:17.631464 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.631469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.632511 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.632512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.633650 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.633652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.634217 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.634218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.634348 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.634883 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:17.634893 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:17.636347 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:17.638291 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.638298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.638300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.638522 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.638745 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.638746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.638746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.638867 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.639129 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:17.639135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:17.639275 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 20:27:17.639389 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 20:27:17.639540 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260619 20:27:17.640303 24054 heartbeater.cc:347] Connected to a master server at 127.21.151.62:35593
I20260619 20:27:17.640386 24054 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:17.640580 24054 heartbeater.cc:511] Master 127.21.151.62:35593 requested a full tablet report, sending...
I20260619 20:27:17.641095 23850 ts_manager.cc:194] Registered new tserver with Master: 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263)
I20260619 20:27:17.641993 23850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:38703
I20260619 20:27:17.642814 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 23923
I20260619 20:27:17.642915 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:17.643092 22108 external_mini_cluster.cc:1428] Setting key 9bf128de5f0faada75cefba29baaa04b
I20260619 20:27:17.647257 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.641483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:52451 (local address 127.21.151.62:35593)
0619 20:27:17.641523 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.641525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.641589 (+ 64us) server_negotiation.cc:407] Connection header received
0619 20:27:17.641653 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.641654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.641661 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.641704 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:17.642033 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.642596 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.643455 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.643762 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.643832 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.644715 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:17.644717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:17.644717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:17.644724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:17.646148 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.646533 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.646533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.646534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.646580 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.646784 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.646785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.646785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.646872 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:17.646919 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.647141 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.647199 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:27:17.647387 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.641395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:35593 (local address 127.21.151.1:52451)
0619 20:27:17.641532 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:17.641543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:17.641601 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:17.641718 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:17.641719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:17.641898 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:17.641957 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.641959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.642683 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.642684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.643375 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.643377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.643848 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.643849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.643929 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.644366 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:17.644370 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:17.644632 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:17.646226 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.646227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.646227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.646459 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.646645 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.646646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.646647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.646719 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.646965 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:17.646966 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:17.647016 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 20:27:17.647120 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 20:27:17.647235 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:27:17.666034 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:35593
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:17.744987 24063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:17.745240 24063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:17.745283 24063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:17.746510 24063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:17.746595 24063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:17.746635 24063 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 20:27:17.746678 24063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:27:17.746719 24063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:17.748384 24063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.151.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:17.748673 24063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:17.748920 24063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:17.754010 24071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:17.754019 24069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:17.754017 24068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:17.754243 24063 server_base.cc:1034] running on GCE node
I20260619 20:27:17.754454 24063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:17.754657 24063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:17.755795 24063 hybrid_clock.cc:648] HybridClock initialized: now 1781900837755769 us; error 39 us; skew 500 ppm
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:17.758383 24063 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:27:17.758863 24063 webserver.cc:460] Webserver started at http://127.21.151.2:38123/ using document root <none> and password file <none>
I20260619 20:27:17.759106 24063 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:17.759153 24063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:17.759323 24063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:17.760461 24063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "66c4a24fb82a4f508f924b0878824024"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "5a7e10dd3c38e545b6147dbf934e1e63"
server_key_iv: "cefb8ed32ffd55a022f9c24c1196bf45"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.760850 24063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "66c4a24fb82a4f508f924b0878824024"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "5a7e10dd3c38e545b6147dbf934e1e63"
server_key_iv: "cefb8ed32ffd55a022f9c24c1196bf45"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.762194 24063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:27:17.762912 24078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:17.763106 24063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:17.763195 24063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "66c4a24fb82a4f508f924b0878824024"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "5a7e10dd3c38e545b6147dbf934e1e63"
server_key_iv: "cefb8ed32ffd55a022f9c24c1196bf45"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.763270 24063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:17.792667 24063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:17.795161 24063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:17.795323 24063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:17.795921 24063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:17.795960 24063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:17.795985 24063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:17.796005 24063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:17.803092 24063 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:35215
I20260619 20:27:17.803169 24193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:35215 every 8 connection(s)
I20260619 20:27:17.803460 24063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:27:17.805858 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.796529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:57061 (local address 127.21.151.62:35593)
0619 20:27:17.796589 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.796591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.796849 (+ 258us) server_negotiation.cc:407] Connection header received
0619 20:27:17.797408 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.797409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.797423 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.797485 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:17.798429 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.799294 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.800446 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.800688 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.800767 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.802922 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:17.802923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:17.802923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:17.802932 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:17.804497 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.804907 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.804908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.804909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.804968 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.805223 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.805224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.805224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.805344 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:17.805420 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.805688 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.805764 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:27:17.806052 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.796529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:35593 (local address 127.21.151.2:57061)
0619 20:27:17.796715 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:17.796753 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:17.797300 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:17.797582 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:17.797587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:17.797977 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:17.798307 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.798314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.799429 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.799430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.800334 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.800336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.800788 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.800789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.800918 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.801456 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:17.801466 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:17.802818 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:17.804602 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.804608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.804610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.804828 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.805040 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.805041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.805042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.805155 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.805412 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:17.805418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:17.805556 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 20:27:17.805667 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 20:27:17.805822 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260619 20:27:17.806559 24194 heartbeater.cc:347] Connected to a master server at 127.21.151.62:35593
I20260619 20:27:17.806646 24194 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:17.806834 24194 heartbeater.cc:511] Master 127.21.151.62:35593 requested a full tablet report, sending...
I20260619 20:27:17.807283 23851 ts_manager.cc:194] Registered new tserver with Master: 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215)
I20260619 20:27:17.807816 23851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:57061
I20260619 20:27:17.810453 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 24063
I20260619 20:27:17.810556 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:17.810751 22108 external_mini_cluster.cc:1428] Setting key 70543af71612cf6f9c3e5795b9643449
I20260619 20:27:17.813675 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.807685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:42831 (local address 127.21.151.62:35593)
0619 20:27:17.807726 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.807728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.807803 (+ 75us) server_negotiation.cc:407] Connection header received
0619 20:27:17.807867 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.807868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.807894 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.807954 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:17.808361 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.808823 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.809554 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.809752 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.809826 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.810935 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:17.810937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:17.810937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:17.810945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:17.812466 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.812844 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.812845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.812845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.812904 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.813117 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.813118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.813118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.813263 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:17.813337 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.813496 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.813570 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:27:17.813674 24200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.807582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:35593 (local address 127.21.151.2:42831)
0619 20:27:17.807742 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:17.807752 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:17.807810 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:17.808062 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:17.808063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:17.808209 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:17.808257 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.808258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.808927 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.808928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.809472 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.809473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.809824 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.809825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.809897 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.810348 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:17.810354 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:17.810795 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:17.812571 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.812572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.812573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.812740 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.812972 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.812972 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.812973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.813029 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.813340 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:17.813342 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:17.813386 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 20:27:17.813471 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:27:17.813571 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:27:17.834621 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:35593
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:17.914333 24203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:17.914523 24203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:17.914547 24203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:17.915680 24203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:17.915719 24203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:17.915737 24203 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 20:27:17.915750 24203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:27:17.915809 24203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:17.917160 24203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.151.62:35593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:17.917385 24203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:17.917629 24203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:17.922572 24208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:17.922608 24209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:17.922564 24211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:17.922899 24203 server_base.cc:1034] running on GCE node
I20260619 20:27:17.923107 24203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:17.923338 24203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:17.924499 24203 hybrid_clock.cc:648] HybridClock initialized: now 1781900837924491 us; error 36 us; skew 500 ppm
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:17.926903 24203 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:27:17.927371 24203 webserver.cc:460] Webserver started at http://127.21.151.3:36219/ using document root <none> and password file <none>
I20260619 20:27:17.927660 24203 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:17.927722 24203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:17.927857 24203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:17.928869 24203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "7f028faf9bbc49c79ceaa0e92ef2859b"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "bddf6811a02e0517d6b55e2c5fa6876b"
server_key_iv: "756868ebb592621bea5150b9e2aeb21c"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.929235 24203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "7f028faf9bbc49c79ceaa0e92ef2859b"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "bddf6811a02e0517d6b55e2c5fa6876b"
server_key_iv: "756868ebb592621bea5150b9e2aeb21c"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.930537 24203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:27:17.931371 24218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:17.931557 24203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:17.931669 24203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "7f028faf9bbc49c79ceaa0e92ef2859b"
format_stamp: "Formatted at 2026-06-19 20:27:17 on dist-test-slave-f9rk"
server_key: "bddf6811a02e0517d6b55e2c5fa6876b"
server_key_iv: "756868ebb592621bea5150b9e2aeb21c"
server_key_version: "encryptionkey@0"
I20260619 20:27:17.931751 24203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:17.938611 24203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:17.940429 24203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:17.940579 24203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:17.941107 24203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:17.941164 24203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:17.941197 24203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:17.941220 24203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:17.948793 24203 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:43985
I20260619 20:27:17.949230 24203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:17.949262 24333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:43985 every 8 connection(s)
I20260619 20:27:17.951354 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.941569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:59551 (local address 127.21.151.62:35593)
0619 20:27:17.941640 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.941643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.941926 (+ 283us) server_negotiation.cc:407] Connection header received
0619 20:27:17.942611 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.942612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.942623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.942687 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:17.943549 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.944114 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.945753 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.945980 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.946072 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.948333 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:17.948334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:17.948334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:17.948343 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:17.949914 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.950387 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.950388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.950388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.950450 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.950698 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.950699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.950699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.950817 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:17.950890 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.951190 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.951265 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:27:17.951522 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.941559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:35593 (local address 127.21.151.3:59551)
0619 20:27:17.941774 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:17.941832 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:17.942494 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:17.942761 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:17.942766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:17.943156 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:17.943446 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.943452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.944185 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.944186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.945667 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.945670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.946096 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.946097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.946225 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.946741 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:17.946749 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:17.948234 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:17.950011 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.950018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.950020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.950298 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.950516 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.950517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.950517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.950630 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.950883 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:17.950889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:17.951030 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 20:27:17.951141 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 20:27:17.951303 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260619 20:27:17.952056 24334 heartbeater.cc:347] Connected to a master server at 127.21.151.62:35593
I20260619 20:27:17.952135 24334 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:17.952316 24334 heartbeater.cc:511] Master 127.21.151.62:35593 requested a full tablet report, sending...
I20260619 20:27:17.952747 23850 ts_manager.cc:194] Registered new tserver with Master: 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985)
I20260619 20:27:17.953169 23850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:59551
I20260619 20:27:17.958163 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.953171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:52195 (local address 127.21.151.62:35593)
0619 20:27:17.953209 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.953211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.953280 (+ 69us) server_negotiation.cc:407] Connection header received
0619 20:27:17.953368 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.953369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.953378 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.953429 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:17.953760 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.954204 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.954861 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.955059 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.955143 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.955990 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:17.955991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:17.955992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:17.955998 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:17.957050 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.957390 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.957391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.957392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.957466 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.957675 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.957676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.957676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.957780 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:17.957850 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.958006 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.958076 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:17.958185 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.953081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:35593 (local address 127.21.151.3:52195)
0619 20:27:17.953215 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:17.953226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:17.953304 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:17.953447 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:17.953447 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:17.953614 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:17.953683 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.953684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.954290 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.954291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.954782 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.954784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.955146 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.955147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.955219 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.955574 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:17.955577 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:17.955898 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:17.957148 (+ 1250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.957149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.957149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.957309 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.957537 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:17.957538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:17.957539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:17.957595 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:17.957854 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:17.957855 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:17.957898 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 20:27:17.957981 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 20:27:17.958079 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":6,"thread_start_us":80,"threads_started":1}
I20260619 20:27:17.959281 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 24203
I20260619 20:27:17.959419 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:17.959656 22108 external_mini_cluster.cc:1428] Setting key 97f5423b8a042f3dfc9f7406758cad41
I20260619 20:27:17.960471 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:27:17 dist-test-slave-f9rk krb5kdc[23806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:17.983029 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.975704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.21.151.62:35593)
0619 20:27:17.975753 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.975755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.975782 (+ 27us) server_negotiation.cc:407] Connection header received
0619 20:27:17.975803 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.975804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.975812 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.975851 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:17.976364 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.976924 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.977677 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.977905 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.978004 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.980547 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:17.980548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:17.980549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:17.980556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:17.981921 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.982275 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.982276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.982277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.982331 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:17.982532 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:17.982533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:17.982533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:17.982640 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:17.982745 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.982851 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.982928 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:27:17.984572 23850 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:17.986079 23850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:17.992280 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.988143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:43985 (local address 127.0.0.1:58044)
0619 20:27:17.988299 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:17.988339 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:17.988398 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:17.988824 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:17.988827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:17.988829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:17.988899 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.988900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.989898 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.989899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.991043 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.991045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.991919 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.991920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.992021 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.992041 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 20:27:17.992147 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 20:27:17.992198 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
I20260619 20:27:17.992511 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.988257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.21.151.1:41263)
0619 20:27:17.988824 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.988828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.988837 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:17.988872 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.988873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.988887 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.989174 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:17.989267 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.990285 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.991285 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.992029 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.992104 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.992176 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.992333 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.992392 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":102,"threads_started":1}
I20260619 20:27:17.992625 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.988143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:41263 (local address 127.0.0.1:57224)
0619 20:27:17.988614 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:17.988624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:17.988700 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:17.989058 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:17.989058 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:17.989060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:17.989098 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.989099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.990344 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.990344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.991208 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.991211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.992135 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.992136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.992188 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.992191 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:17.992239 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:27:17.992269 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260619 20:27:17.992641 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.988455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.21.151.3:43985)
0619 20:27:17.988617 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.988622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.988634 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:27:17.988692 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.988694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.988710 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.989017 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:17.989078 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.989810 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.991133 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.991826 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.991916 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.991975 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.992499 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.992544 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260619 20:27:17.992836 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.988326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:35215 (local address 127.0.0.1:44730)
0619 20:27:17.988622 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:17.988632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:17.988707 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:17.988958 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:17.988959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:17.988961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:17.989015 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.989017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.990300 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.990301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.991372 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:17.991374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.992244 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.992245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:17.992304 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.992309 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:27:17.992618 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 20:27:17.992664 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":25,"thread_start_us":56,"threads_started":1}
I20260619 20:27:17.993005 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:17.988444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.21.151.2:35215)
0619 20:27:17.988602 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:17.988608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:17.988716 (+ 108us) server_negotiation.cc:407] Connection header received
0619 20:27:17.988779 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:17.988782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:17.988798 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:17.988876 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:17.989098 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.990211 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.991464 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:17.992171 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:17.992231 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:17.992279 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:17.992477 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 20:27:17.992514 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260619 20:27:17.993302 24268 tablet_service.cc:1467] Processing CreateTablet for tablet d444bef857924f55ac25113c53e9460a (DEFAULT_TABLE table=test-table [id=e8fbb15ccde140e69966ff8eb867ebb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:17.993306 23988 tablet_service.cc:1467] Processing CreateTablet for tablet d444bef857924f55ac25113c53e9460a (DEFAULT_TABLE table=test-table [id=e8fbb15ccde140e69966ff8eb867ebb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:17.993652 23988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d444bef857924f55ac25113c53e9460a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:17.993655 24268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d444bef857924f55ac25113c53e9460a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:17.996182 24354 tablet_bootstrap.cc:492] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: Bootstrap starting.
I20260619 20:27:17.996795 24355 tablet_bootstrap.cc:492] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: Bootstrap starting.
I20260619 20:27:17.996794 24128 tablet_service.cc:1467] Processing CreateTablet for tablet d444bef857924f55ac25113c53e9460a (DEFAULT_TABLE table=test-table [id=e8fbb15ccde140e69966ff8eb867ebb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:17.997100 24128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d444bef857924f55ac25113c53e9460a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:17.997221 24354 tablet_bootstrap.cc:654] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:17.997382 24355 tablet_bootstrap.cc:654] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:17.997570 24354 log.cc:824] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:17.997738 24355 log.cc:824] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:17.998533 24354 tablet_bootstrap.cc:492] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: No bootstrap required, opened a new log
I20260619 20:27:17.998592 24354 ts_tablet_manager.cc:1397] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:27:17.998826 24355 tablet_bootstrap.cc:492] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: No bootstrap required, opened a new log
I20260619 20:27:17.998950 24355 ts_tablet_manager.cc:1397] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:27:18.000123 24354 raft_consensus.cc:348] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.000252 24354 raft_consensus.cc:374] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:18.000274 24354 raft_consensus.cc:729] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f028faf9bbc49c79ceaa0e92ef2859b, State: Initialized, Role: FOLLOWER
I20260619 20:27:18.000391 24354 consensus_queue.cc:260] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.000671 24354 ts_tablet_manager.cc:1428] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:27:18.000762 24334 heartbeater.cc:503] Master 127.21.151.62:35593 was elected leader, sending a full tablet report...
I20260619 20:27:18.001153 24359 tablet_bootstrap.cc:492] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: Bootstrap starting.
I20260619 20:27:18.001729 24359 tablet_bootstrap.cc:654] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:18.001669 24355 raft_consensus.cc:348] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.001834 24355 raft_consensus.cc:374] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:18.001874 24355 raft_consensus.cc:729] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f4a0f200a6437a9a734e071b535003, State: Initialized, Role: FOLLOWER
I20260619 20:27:18.001979 24355 consensus_queue.cc:260] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.002045 24359 log.cc:824] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:18.002260 24355 ts_tablet_manager.cc:1428] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:27:18.002300 24054 heartbeater.cc:503] Master 127.21.151.62:35593 was elected leader, sending a full tablet report...
I20260619 20:27:18.003499 24359 tablet_bootstrap.cc:492] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: No bootstrap required, opened a new log
I20260619 20:27:18.003569 24359 ts_tablet_manager.cc:1397] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:27:18.005985 24359 raft_consensus.cc:348] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.006163 24359 raft_consensus.cc:374] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:18.006196 24359 raft_consensus.cc:729] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c4a24fb82a4f508f924b0878824024, State: Initialized, Role: FOLLOWER
I20260619 20:27:18.006319 24359 consensus_queue.cc:260] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.006664 24359 ts_tablet_manager.cc:1428] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:27:18.006800 24194 heartbeater.cc:503] Master 127.21.151.62:35593 was elected leader, sending a full tablet report...
W20260619 20:27:18.054304 24195 tablet.cc:2369] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:27:18.138278 24055 tablet.cc:2369] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:27:18.199843 24335 tablet.cc:2369] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:27:18.257277 24358 raft_consensus.cc:482] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:18.257434 24358 raft_consensus.cc:504] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.257845 24358 leader_election.cc:288] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263), 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215)
I20260619 20:27:18.260677 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.258045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:41263 (local address 127.21.151.3:45619)
0619 20:27:18.258119 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:18.258133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:18.258194 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:18.258332 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:18.258333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:18.258334 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:18.258409 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.258411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.259064 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.259064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.259837 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.259839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.260388 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.260389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.260475 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.260478 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:18.260547 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:27:18.260589 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 20:27:18.260677 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.258119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:45619 (local address 127.21.151.1:41263)
0619 20:27:18.258194 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.258197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.258209 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:27:18.258249 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.258250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.258261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.258321 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:18.258498 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.258977 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.259936 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.260306 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.260392 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.260401 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.260553 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.260594 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 20:27:18.260943 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.258279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:56639 (local address 127.21.151.2:35215)
0619 20:27:18.258324 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.258326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.258378 (+ 52us) server_negotiation.cc:407] Connection header received
0619 20:27:18.258457 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.258458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.258468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.258529 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:18.258698 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.259172 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.260176 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.260534 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.260641 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.260648 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.260835 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.260878 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:27:18.261055 24008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d444bef857924f55ac25113c53e9460a" candidate_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f4a0f200a6437a9a734e071b535003" is_pre_election: true
I20260619 20:27:18.261144 24362 raft_consensus.cc:482] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:18.261181 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.258169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:35215 (local address 127.21.151.3:56639)
0619 20:27:18.258326 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:18.258335 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:18.258384 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:18.258546 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:18.258547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:18.258549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:18.258621 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.258623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.259260 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.259262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.260093 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.260095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.260640 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.260640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.260718 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.260721 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:18.261030 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 20:27:18.261078 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260619 20:27:18.261216 24008 raft_consensus.cc:2454] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f028faf9bbc49c79ceaa0e92ef2859b in term 0.
I20260619 20:27:18.261243 24362 raft_consensus.cc:504] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.261462 24220 leader_election.cc:302] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f4a0f200a6437a9a734e071b535003, 7f028faf9bbc49c79ceaa0e92ef2859b; no voters:
I20260619 20:27:18.261598 24358 raft_consensus.cc:2781] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:18.261575 24148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d444bef857924f55ac25113c53e9460a" candidate_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c4a24fb82a4f508f924b0878824024" is_pre_election: true
I20260619 20:27:18.261642 24358 raft_consensus.cc:482] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:18.261664 24358 raft_consensus.cc:3037] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:18.261669 24148 raft_consensus.cc:2454] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f028faf9bbc49c79ceaa0e92ef2859b in term 0.
I20260619 20:27:18.262121 24362 leader_election.cc:288] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263), 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985)
I20260619 20:27:18.262647 24358 raft_consensus.cc:504] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.262761 24358 leader_election.cc:288] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [CANDIDATE]: Term 1 election: Requested vote from peers 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263), 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215)
I20260619 20:27:18.263028 24148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d444bef857924f55ac25113c53e9460a" candidate_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c4a24fb82a4f508f924b0878824024"
I20260619 20:27:18.263105 24148 raft_consensus.cc:3037] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:18.264004 24148 raft_consensus.cc:2454] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f028faf9bbc49c79ceaa0e92ef2859b in term 1.
I20260619 20:27:18.264106 24008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d444bef857924f55ac25113c53e9460a" candidate_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f4a0f200a6437a9a734e071b535003"
I20260619 20:27:18.264182 24008 raft_consensus.cc:3037] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:18.265031 24008 raft_consensus.cc:2454] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f028faf9bbc49c79ceaa0e92ef2859b in term 1.
I20260619 20:27:18.265156 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.261802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:45669 (local address 127.21.151.1:41263)
0619 20:27:18.261837 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.261838 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.262064 (+ 226us) server_negotiation.cc:407] Connection header received
0619 20:27:18.262097 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.262098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.262107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.262152 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:18.262309 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.262946 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.264283 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.264780 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.264848 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.264856 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.265058 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.265095 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:27:18.265164 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.261761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:41263 (local address 127.21.151.2:45669)
0619 20:27:18.261853 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:18.261863 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:18.261901 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:18.262191 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:18.262191 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:18.262193 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:18.262250 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.262252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.263030 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.263031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.264188 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.264190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.264889 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.264890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.264973 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.264977 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:18.265039 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:27:18.265087 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 20:27:18.265383 24220 leader_election.cc:302] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c4a24fb82a4f508f924b0878824024, 7f028faf9bbc49c79ceaa0e92ef2859b; no voters:
I20260619 20:27:18.265378 24008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d444bef857924f55ac25113c53e9460a" candidate_uuid: "66c4a24fb82a4f508f924b0878824024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f4a0f200a6437a9a734e071b535003" is_pre_election: true
I20260619 20:27:18.265463 24008 raft_consensus.cc:2379] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66c4a24fb82a4f508f924b0878824024 in current term 1: Already voted for candidate 7f028faf9bbc49c79ceaa0e92ef2859b in this term.
I20260619 20:27:18.265528 24358 raft_consensus.cc:2781] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:18.265648 24358 raft_consensus.cc:686] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 LEADER]: Becoming Leader. State: Replica: 7f028faf9bbc49c79ceaa0e92ef2859b, State: Running, Role: LEADER
I20260619 20:27:18.265754 24358 consensus_queue.cc:237] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.265805 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.261761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:43985 (local address 127.21.151.2:58971)
0619 20:27:18.261918 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:18.261929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:18.261981 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:18.262094 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:18.262095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:18.262096 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:18.262152 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.262153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.263491 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.263492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.264757 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.264759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.265409 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.265410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.265493 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.265497 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:18.265668 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 20:27:18.265711 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":10,"thread_start_us":97,"threads_started":1}
I20260619 20:27:18.266240 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.261962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:58971 (local address 127.21.151.3:43985)
0619 20:27:18.261998 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.262000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.262007 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:27:18.262029 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.262029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.262037 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.262089 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:18.262243 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.263377 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.264842 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.265307 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.265408 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.265416 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.265597 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.265631 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:18.266502 24288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d444bef857924f55ac25113c53e9460a" candidate_uuid: "66c4a24fb82a4f508f924b0878824024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" is_pre_election: true
I20260619 20:27:18.266624 23850 catalog_manager.cc:5495] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3). New cstate: current_term: 1 leader_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } health_report { overall_health: HEALTHY } } }
I20260619 20:27:18.266824 24081 leader_election.cc:302] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66c4a24fb82a4f508f924b0878824024; no voters: 55f4a0f200a6437a9a734e071b535003, 7f028faf9bbc49c79ceaa0e92ef2859b
I20260619 20:27:18.266974 24362 raft_consensus.cc:2726] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Jun 19 20:27:18 dist-test-slave-f9rk krb5kdc[23806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900837, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:18.297978 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.288385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.21.151.62:35593)
0619 20:27:18.288460 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.288462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.288529 (+ 67us) server_negotiation.cc:407] Connection header received
0619 20:27:18.288576 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.288576 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.288591 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.288642 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:18.289082 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.289552 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.290225 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.290395 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.290444 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.295683 (+ 5239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:18.295685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:18.295685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:18.295691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:18.297072 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:18.297382 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:18.297383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:18.297383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:18.297430 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:18.297603 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:18.297603 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:18.297604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:18.297698 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:18.297745 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.297853 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.297915 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:27:18.298137 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.288305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:35593 (local address 127.0.0.1:60532)
0619 20:27:18.288435 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:18.288447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:18.288533 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:18.288671 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:18.288671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:18.288960 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:18.289011 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.289013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.289630 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.289630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.290159 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.290160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.290458 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.290459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.290504 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.294188 (+ 3684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:18.294193 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:18.295556 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:18.297140 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:18.297146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:18.297147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:18.297322 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:18.297489 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:18.297490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:18.297490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:18.297547 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:18.297749 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:18.297754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:18.297792 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 20:27:18.297840 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:27:18.297980 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":56,"thread_start_us":60,"threads_started":1}
I20260619 20:27:18.300068 23850 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:60532:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 20:27:18.300295 23850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:18.301822 24268 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9fe7ab1cae4e8eae38d33174102c5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=027c50e84b834c4daf35d704b5050746]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:27:18.301822 23988 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9fe7ab1cae4e8eae38d33174102c5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=027c50e84b834c4daf35d704b5050746]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:27:18.301844 24128 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9fe7ab1cae4e8eae38d33174102c5c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=027c50e84b834c4daf35d704b5050746]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:27:18.301994 24268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9fe7ab1cae4e8eae38d33174102c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:18.301998 23988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9fe7ab1cae4e8eae38d33174102c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:18.302008 24128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9fe7ab1cae4e8eae38d33174102c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:18.303130 24359 tablet_bootstrap.cc:492] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: Bootstrap starting.
I20260619 20:27:18.303508 24354 tablet_bootstrap.cc:492] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: Bootstrap starting.
I20260619 20:27:18.303560 24359 tablet_bootstrap.cc:654] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:18.303943 24354 tablet_bootstrap.cc:654] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:18.304277 24355 tablet_bootstrap.cc:492] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: Bootstrap starting.
I20260619 20:27:18.304301 24359 tablet_bootstrap.cc:492] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: No bootstrap required, opened a new log
I20260619 20:27:18.304335 24359 ts_tablet_manager.cc:1397] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:18.304477 24359 raft_consensus.cc:348] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.304548 24354 tablet_bootstrap.cc:492] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: No bootstrap required, opened a new log
I20260619 20:27:18.304551 24359 raft_consensus.cc:374] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:18.304581 24354 ts_tablet_manager.cc:1397] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:18.304579 24359 raft_consensus.cc:729] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c4a24fb82a4f508f924b0878824024, State: Initialized, Role: FOLLOWER
I20260619 20:27:18.304649 24355 tablet_bootstrap.cc:654] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:18.304646 24359 consensus_queue.cc:260] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.304759 24359 ts_tablet_manager.cc:1428] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:18.304729 24354 raft_consensus.cc:348] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.304802 24354 raft_consensus.cc:374] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:18.304842 24354 raft_consensus.cc:729] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f028faf9bbc49c79ceaa0e92ef2859b, State: Initialized, Role: FOLLOWER
I20260619 20:27:18.304884 24354 consensus_queue.cc:260] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.304909 24362 tablet_replica.cc:439] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } } }
I20260619 20:27:18.304972 24354 ts_tablet_manager.cc:1428] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:18.304982 24362 tablet_replica.cc:442] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:18.305032 24365 tablet_replica.cc:439] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } } }
I20260619 20:27:18.305122 24365 tablet_replica.cc:442] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:18.305593 24355 tablet_bootstrap.cc:492] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: No bootstrap required, opened a new log
I20260619 20:27:18.305648 24355 ts_tablet_manager.cc:1397] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:18.305796 24355 raft_consensus.cc:348] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.305877 24355 raft_consensus.cc:374] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:18.305900 24355 raft_consensus.cc:729] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f4a0f200a6437a9a734e071b535003, State: Initialized, Role: FOLLOWER
I20260619 20:27:18.305941 24355 consensus_queue.cc:260] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.306032 24355 ts_tablet_manager.cc:1428] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:18.306099 24360 tablet_replica.cc:439] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } } }
I20260619 20:27:18.306211 24360 tablet_replica.cc:442] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:18.345248 24360 raft_consensus.cc:482] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:18.345338 24360 raft_consensus.cc:504] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.345633 24360 leader_election.cc:288] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215), 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985)
I20260619 20:27:18.349629 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.345868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:40001 (local address 127.21.151.2:35215)
0619 20:27:18.345915 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.345918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.346031 (+ 113us) server_negotiation.cc:407] Connection header received
0619 20:27:18.346072 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.346072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.346081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.346148 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:18.346342 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.347098 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.348727 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.349253 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.349318 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.349326 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.349521 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.349557 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:27:18.349632 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.345879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:47989 (local address 127.21.151.3:43985)
0619 20:27:18.345919 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.345921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.346031 (+ 110us) server_negotiation.cc:407] Connection header received
0619 20:27:18.346069 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.346069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.346080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.346148 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:18.346343 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.347088 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.348678 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.349236 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.349318 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.349326 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.349522 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.349559 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:18.349671 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.345820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:35215 (local address 127.21.151.1:40001)
0619 20:27:18.345936 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:18.345947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:18.346018 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:18.346166 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:18.346167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:18.346169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:18.346269 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.346270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.347183 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.347184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.348650 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.348652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.349342 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.349343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.349439 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.349444 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:27:18.349503 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:27:18.349556 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":12,"thread_start_us":57,"threads_started":1}
I20260619 20:27:18.349674 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.345820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:43985 (local address 127.21.151.1:47989)
0619 20:27:18.345935 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:18.345947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:18.346018 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:18.346166 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:18.346167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:18.346171 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:18.346274 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.346275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.347181 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.347182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.348594 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.348597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.349351 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.349352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.349439 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.349443 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:18.349511 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:27:18.349556 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":30,"thread_start_us":57,"threads_started":1}
I20260619 20:27:18.349994 24288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9fe7ab1cae4e8eae38d33174102c5c" candidate_uuid: "55f4a0f200a6437a9a734e071b535003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" is_pre_election: true
I20260619 20:27:18.350001 24148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9fe7ab1cae4e8eae38d33174102c5c" candidate_uuid: "55f4a0f200a6437a9a734e071b535003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c4a24fb82a4f508f924b0878824024" is_pre_election: true
I20260619 20:27:18.350102 24288 raft_consensus.cc:2454] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55f4a0f200a6437a9a734e071b535003 in term 0.
I20260619 20:27:18.350119 24148 raft_consensus.cc:2454] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55f4a0f200a6437a9a734e071b535003 in term 0.
I20260619 20:27:18.350276 23941 leader_election.cc:302] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f4a0f200a6437a9a734e071b535003, 7f028faf9bbc49c79ceaa0e92ef2859b; no voters:
I20260619 20:27:18.350423 24360 raft_consensus.cc:2781] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:18.350476 24360 raft_consensus.cc:482] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:18.350497 24360 raft_consensus.cc:3037] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:18.351217 24360 raft_consensus.cc:504] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.351361 24360 leader_election.cc:288] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [CANDIDATE]: Term 1 election: Requested vote from peers 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215), 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985)
I20260619 20:27:18.351509 24148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9fe7ab1cae4e8eae38d33174102c5c" candidate_uuid: "55f4a0f200a6437a9a734e071b535003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c4a24fb82a4f508f924b0878824024"
I20260619 20:27:18.351538 24288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9fe7ab1cae4e8eae38d33174102c5c" candidate_uuid: "55f4a0f200a6437a9a734e071b535003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b"
I20260619 20:27:18.351585 24148 raft_consensus.cc:3037] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:18.351629 24288 raft_consensus.cc:3037] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:18.352365 24148 raft_consensus.cc:2454] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55f4a0f200a6437a9a734e071b535003 in term 1.
I20260619 20:27:18.352398 24288 raft_consensus.cc:2454] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55f4a0f200a6437a9a734e071b535003 in term 1.
I20260619 20:27:18.352545 23941 leader_election.cc:302] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f4a0f200a6437a9a734e071b535003, 7f028faf9bbc49c79ceaa0e92ef2859b; no voters:
I20260619 20:27:18.352639 24360 raft_consensus.cc:2781] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:18.352752 24360 raft_consensus.cc:686] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [term 1 LEADER]: Becoming Leader. State: Replica: 55f4a0f200a6437a9a734e071b535003, State: Running, Role: LEADER
I20260619 20:27:18.352878 24360 consensus_queue.cc:237] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.353320 24378 tablet_replica.cc:439] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: TxnStatusTablet state changed. Reason: New leader 55f4a0f200a6437a9a734e071b535003. Latest consensus state: current_term: 1 leader_uuid: "55f4a0f200a6437a9a734e071b535003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } } }
I20260619 20:27:18.353417 24378 tablet_replica.cc:442] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:27:18.353689 24380 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 20:27:18.353803 24380 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 20:27:18.353721 23850 catalog_manager.cc:5495] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55f4a0f200a6437a9a734e071b535003 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "55f4a0f200a6437a9a734e071b535003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } health_report { overall_health: UNKNOWN } } }
I20260619 20:27:18.365628 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.362656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.21.151.1:41263)
0619 20:27:18.362694 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.362696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.362819 (+ 123us) server_negotiation.cc:407] Connection header received
0619 20:27:18.362853 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.362854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.362875 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.362937 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:18.363103 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.363762 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.364868 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.365057 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.365109 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.365306 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:18.365392 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:18.365447 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.365514 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.365532 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:18.365654 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.362606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:41263 (local address 127.0.0.1:57232)
0619 20:27:18.362691 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:18.362703 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:18.362751 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:18.362977 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:18.362979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:18.362980 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:18.363033 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.363035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.363847 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.363848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.364786 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.364788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.365128 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.365129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.365201 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.365217 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:18.365449 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:18.365457 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 20:27:18.365511 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:27:18.365565 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 20:27:18.367442 24288 raft_consensus.cc:1261] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Refusing update from remote peer 55f4a0f200a6437a9a734e071b535003: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:18.367753 24148 raft_consensus.cc:1261] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Refusing update from remote peer 55f4a0f200a6437a9a734e071b535003: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:18.367761 24360 consensus_queue.cc:1035] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:18.368002 24360 consensus_queue.cc:1035] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:18.369108 24365 tablet_replica.cc:439] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: TxnStatusTablet state changed. Reason: New leader 55f4a0f200a6437a9a734e071b535003. Latest consensus state: current_term: 1 leader_uuid: "55f4a0f200a6437a9a734e071b535003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } } }
I20260619 20:27:18.369197 24365 tablet_replica.cc:442] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:18.370187 24362 tablet_replica.cc:439] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: TxnStatusTablet state changed. Reason: New leader 55f4a0f200a6437a9a734e071b535003. Latest consensus state: current_term: 1 leader_uuid: "55f4a0f200a6437a9a734e071b535003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } } }
I20260619 20:27:18.370278 24362 tablet_replica.cc:442] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:18.370860 24360 tablet_replica.cc:439] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55f4a0f200a6437a9a734e071b535003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } } }
I20260619 20:27:18.370983 24360 tablet_replica.cc:442] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:27:18.370985 24365 tablet_replica.cc:439] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55f4a0f200a6437a9a734e071b535003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } } }
I20260619 20:27:18.371047 24365 tablet_replica.cc:442] T 9e9fe7ab1cae4e8eae38d33174102c5c P 7f028faf9bbc49c79ceaa0e92ef2859b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:18.371085 24360 tablet_replica.cc:439] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55f4a0f200a6437a9a734e071b535003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } } }
I20260619 20:27:18.371117 24384 mvcc.cc:205] Tried to move back new op lower bound from 7298665833950515200 to 7298665833893990400. Current Snapshot: MvccSnapshot[applied={T|T < 7298665833950515200}]
I20260619 20:27:18.371146 24360 tablet_replica.cc:442] T 9e9fe7ab1cae4e8eae38d33174102c5c P 55f4a0f200a6437a9a734e071b535003: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:27:18.371922 24362 tablet_replica.cc:439] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55f4a0f200a6437a9a734e071b535003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } } }
I20260619 20:27:18.372006 24362 tablet_replica.cc:442] T 9e9fe7ab1cae4e8eae38d33174102c5c P 66c4a24fb82a4f508f924b0878824024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:18.372298 24383 mvcc.cc:205] Tried to move back new op lower bound from 7298665833950515200 to 7298665833893990400. Current Snapshot: MvccSnapshot[applied={T|T < 7298665833950515200}]
I20260619 20:27:18.380007 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.377551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.21.151.3:43985)
0619 20:27:18.377611 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.377613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.377622 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:18.377654 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.377655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.377674 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.377769 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:18.377924 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.378406 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.379163 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.379373 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.379475 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.379673 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:18.379782 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:18.379873 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.379895 (+ 22us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.379917 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:27:18.385083 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.382511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:58645 (local address 127.21.151.1:41263)
0619 20:27:18.382574 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.382576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.382630 (+ 54us) server_negotiation.cc:407] Connection header received
0619 20:27:18.382701 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.382702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.382709 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.382753 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:18.382890 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.383403 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.384470 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.384643 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.384693 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.384897 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:18.384921 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:18.384956 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.385016 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.385032 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:27:18.385126 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.382439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:41263 (local address 127.21.151.3:58645)
0619 20:27:18.382573 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:18.382593 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:18.382644 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:18.382774 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:18.382775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:18.382776 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:18.382830 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.382831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.383480 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.383481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.384389 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.384392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.384719 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.384720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.384788 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.384803 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:18.384962 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:18.384964 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:27:18.385012 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:27:18.385054 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 20:27:18.387267 24148 raft_consensus.cc:1261] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Refusing update from remote peer 7f028faf9bbc49c79ceaa0e92ef2859b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:18.387352 24008 raft_consensus.cc:1261] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Refusing update from remote peer 7f028faf9bbc49c79ceaa0e92ef2859b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:18.387607 24358 consensus_queue.cc:1035] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:18.387758 24365 consensus_queue.cc:1035] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:18.396062 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.393246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:59477 (local address 127.21.151.3:43985)
0619 20:27:18.393283 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:18.393285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:18.393427 (+ 142us) server_negotiation.cc:407] Connection header received
0619 20:27:18.393485 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:18.393486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:18.393495 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:18.393549 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:18.393693 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.394138 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.395272 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.395485 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.395556 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.395817 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:18.395851 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:18.395894 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:18.395952 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 20:27:18.395972 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:27:18.396521 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:18.393200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:43985 (local address 127.21.151.1:59477)
0619 20:27:18.393357 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:18.393366 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:18.393437 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:18.393554 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:18.393554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:18.393556 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:18.393616 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.393617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.394217 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.394218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.395192 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:18.395194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:18.395568 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:18.395569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:18.395704 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:18.395719 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:18.395905 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:18.395907 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:27:18.395950 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 20:27:18.396403 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260619 20:27:18.409650 23851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:18.409780 23851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:18.410528 23851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94: Sending DeleteTablet for 3 replicas of tablet d444bef857924f55ac25113c53e9460a
I20260619 20:27:18.410856 23988 tablet_service.cc:1514] Processing DeleteTablet for tablet d444bef857924f55ac25113c53e9460a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:18 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57224
I20260619 20:27:18.410938 24268 tablet_service.cc:1514] Processing DeleteTablet for tablet d444bef857924f55ac25113c53e9460a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:18 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:58044
I20260619 20:27:18.411247 24128 tablet_service.cc:1514] Processing DeleteTablet for tablet d444bef857924f55ac25113c53e9460a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:18 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:44730
I20260619 20:27:18.411360 23851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:18.411527 23851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:18.411924 24403 tablet_replica.cc:330] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: stopping tablet replica
I20260619 20:27:18.412034 24403 raft_consensus.cc:2229] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:18.412134 24403 raft_consensus.cc:2258] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:18.412936 24403 ts_tablet_manager.cc:1905] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:18.413014 24128 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef2e81709314e62bcd6fca7fa97b70d (DEFAULT_TABLE table=test-table [id=8da3082782b2405ead5e3c4728281994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:18.413138 24128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef2e81709314e62bcd6fca7fa97b70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:18.414572 24359 tablet_bootstrap.cc:492] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024: Bootstrap starting.
I20260619 20:27:18.414824 24403 ts_tablet_manager.cc:1918] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:18.414897 24403 log.cc:1196] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/d444bef857924f55ac25113c53e9460a
I20260619 20:27:18.414949 24268 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef2e81709314e62bcd6fca7fa97b70d (DEFAULT_TABLE table=test-table [id=8da3082782b2405ead5e3c4728281994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:18.415005 24359 tablet_bootstrap.cc:654] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:18.415068 24268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef2e81709314e62bcd6fca7fa97b70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:18.415169 24403 ts_tablet_manager.cc:1939] T d444bef857924f55ac25113c53e9460a P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting consensus metadata
I20260619 20:27:18.415750 23837 catalog_manager.cc:4841] TS 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985): tablet d444bef857924f55ac25113c53e9460a (table test-table [id=e8fbb15ccde140e69966ff8eb867ebb4]) successfully deleted
I20260619 20:27:18.416455 24354 tablet_bootstrap.cc:492] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b: Bootstrap starting.
I20260619 20:27:18.416548 24404 tablet_replica.cc:330] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: stopping tablet replica
I20260619 20:27:18.412952 23988 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef2e81709314e62bcd6fca7fa97b70d (DEFAULT_TABLE table=test-table [id=8da3082782b2405ead5e3c4728281994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:18.416733 23988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef2e81709314e62bcd6fca7fa97b70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:18.416868 24354 tablet_bootstrap.cc:654] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:18.417570 24354 tablet_bootstrap.cc:492] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b: No bootstrap required, opened a new log
I20260619 20:27:18.417613 24354 ts_tablet_manager.cc:1397] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:18.417749 24354 raft_consensus.cc:348] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.417814 24354 raft_consensus.cc:374] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:18.417832 24354 raft_consensus.cc:729] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f028faf9bbc49c79ceaa0e92ef2859b, State: Initialized, Role: FOLLOWER
I20260619 20:27:18.417872 24354 consensus_queue.cc:260] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.417953 24354 ts_tablet_manager.cc:1428] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:18.418325 24355 tablet_bootstrap.cc:492] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003: Bootstrap starting.
I20260619 20:27:18.418706 24355 tablet_bootstrap.cc:654] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:18.419425 24355 tablet_bootstrap.cc:492] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003: No bootstrap required, opened a new log
I20260619 20:27:18.419500 24355 ts_tablet_manager.cc:1397] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:18.419692 24355 raft_consensus.cc:348] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.419773 24355 raft_consensus.cc:374] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:18.419801 24355 raft_consensus.cc:729] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f4a0f200a6437a9a734e071b535003, State: Initialized, Role: FOLLOWER
I20260619 20:27:18.419843 24355 consensus_queue.cc:260] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.419955 24355 ts_tablet_manager.cc:1428] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:18.411823 24402 tablet_replica.cc:330] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: stopping tablet replica
I20260619 20:27:18.420105 24359 tablet_bootstrap.cc:492] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024: No bootstrap required, opened a new log
I20260619 20:27:18.420297 24359 ts_tablet_manager.cc:1397] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260619 20:27:18.420449 24359 raft_consensus.cc:348] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.420528 24359 raft_consensus.cc:374] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:18.420553 24359 raft_consensus.cc:729] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c4a24fb82a4f508f924b0878824024, State: Initialized, Role: FOLLOWER
I20260619 20:27:18.420598 24359 consensus_queue.cc:260] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.420725 24359 ts_tablet_manager.cc:1428] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:18.421187 24402 raft_consensus.cc:2229] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:18.421386 24404 raft_consensus.cc:2229] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:18.421489 24404 raft_consensus.cc:2258] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:18.421259 24402 raft_consensus.cc:2258] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:18.421993 24402 ts_tablet_manager.cc:1905] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:18.423192 24404 ts_tablet_manager.cc:1905] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:18.423208 24402 ts_tablet_manager.cc:1918] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:18.423362 24402 log.cc:1196] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/d444bef857924f55ac25113c53e9460a
I20260619 20:27:18.423980 24402 ts_tablet_manager.cc:1939] T d444bef857924f55ac25113c53e9460a P 55f4a0f200a6437a9a734e071b535003: Deleting consensus metadata
I20260619 20:27:18.424402 23836 catalog_manager.cc:4841] TS 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263): tablet d444bef857924f55ac25113c53e9460a (table test-table [id=e8fbb15ccde140e69966ff8eb867ebb4]) successfully deleted
I20260619 20:27:18.424896 24404 ts_tablet_manager.cc:1918] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:18.424988 24404 log.cc:1196] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/d444bef857924f55ac25113c53e9460a
I20260619 20:27:18.425213 24404 ts_tablet_manager.cc:1939] T d444bef857924f55ac25113c53e9460a P 66c4a24fb82a4f508f924b0878824024: Deleting consensus metadata
I20260619 20:27:18.425591 23836 catalog_manager.cc:4841] TS 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215): tablet d444bef857924f55ac25113c53e9460a (table test-table [id=e8fbb15ccde140e69966ff8eb867ebb4]) successfully deleted
I20260619 20:27:18.504580 24400 raft_consensus.cc:482] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:18.504694 24400 raft_consensus.cc:504] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.504849 24400 leader_election.cc:288] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263), 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215)
I20260619 20:27:18.505096 24008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef2e81709314e62bcd6fca7fa97b70d" candidate_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f4a0f200a6437a9a734e071b535003" is_pre_election: true
I20260619 20:27:18.505193 24008 raft_consensus.cc:2454] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f028faf9bbc49c79ceaa0e92ef2859b in term 0.
I20260619 20:27:18.505138 24148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef2e81709314e62bcd6fca7fa97b70d" candidate_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c4a24fb82a4f508f924b0878824024" is_pre_election: true
I20260619 20:27:18.505245 24148 raft_consensus.cc:2454] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f028faf9bbc49c79ceaa0e92ef2859b in term 0.
I20260619 20:27:18.505365 24220 leader_election.cc:302] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f4a0f200a6437a9a734e071b535003, 7f028faf9bbc49c79ceaa0e92ef2859b; no voters:
I20260619 20:27:18.505450 24400 raft_consensus.cc:2781] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:18.505483 24400 raft_consensus.cc:482] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:18.505504 24400 raft_consensus.cc:3037] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:18.506494 24400 raft_consensus.cc:504] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.506623 24400 leader_election.cc:288] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [CANDIDATE]: Term 1 election: Requested vote from peers 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263), 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215)
I20260619 20:27:18.506908 24148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef2e81709314e62bcd6fca7fa97b70d" candidate_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c4a24fb82a4f508f924b0878824024"
I20260619 20:27:18.506933 24008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef2e81709314e62bcd6fca7fa97b70d" candidate_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f4a0f200a6437a9a734e071b535003"
I20260619 20:27:18.506994 24148 raft_consensus.cc:3037] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:18.507007 24008 raft_consensus.cc:3037] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:18.507791 24148 raft_consensus.cc:2454] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f028faf9bbc49c79ceaa0e92ef2859b in term 1.
I20260619 20:27:18.507839 24008 raft_consensus.cc:2454] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f028faf9bbc49c79ceaa0e92ef2859b in term 1.
I20260619 20:27:18.507951 24220 leader_election.cc:302] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c4a24fb82a4f508f924b0878824024, 7f028faf9bbc49c79ceaa0e92ef2859b; no voters:
I20260619 20:27:18.508086 24400 raft_consensus.cc:2781] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:18.508143 24400 raft_consensus.cc:686] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 LEADER]: Becoming Leader. State: Replica: 7f028faf9bbc49c79ceaa0e92ef2859b, State: Running, Role: LEADER
I20260619 20:27:18.508189 24400 consensus_queue.cc:237] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:18.508711 23850 catalog_manager.cc:5495] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3). New cstate: current_term: 1 leader_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } health_report { overall_health: HEALTHY } } }
I20260619 20:27:18.535684 24148 raft_consensus.cc:1261] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Refusing update from remote peer 7f028faf9bbc49c79ceaa0e92ef2859b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:18.535861 24008 raft_consensus.cc:1261] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Refusing update from remote peer 7f028faf9bbc49c79ceaa0e92ef2859b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:18.536108 24365 consensus_queue.cc:1035] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:18.536223 24400 consensus_queue.cc:1035] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [LEADER]: Connected to new peer: Peer: permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:18.538096 24386 mvcc.cc:205] Tried to move back new op lower bound from 7298665834640785408 to 7298665834529882112. Current Snapshot: MvccSnapshot[applied={T|T < 7298665834640785408}]
W20260619 20:27:18.541353 24384 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 20:27:20.062357 23851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:20.062467 23851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:20.063323 23851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94: Sending DeleteTablet for 3 replicas of tablet 9ef2e81709314e62bcd6fca7fa97b70d
I20260619 20:27:20.063560 24268 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef2e81709314e62bcd6fca7fa97b70d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:20 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:58044
I20260619 20:27:20.063642 23988 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef2e81709314e62bcd6fca7fa97b70d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:20 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57224
I20260619 20:27:20.063637 24128 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef2e81709314e62bcd6fca7fa97b70d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:20 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:44730
I20260619 20:27:20.063854 24435 tablet_replica.cc:330] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024: stopping tablet replica
I20260619 20:27:20.063923 24435 raft_consensus.cc:2229] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:20.063958 24435 raft_consensus.cc:2258] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:20.064055 24434 tablet_replica.cc:330] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003: stopping tablet replica
I20260619 20:27:20.064118 24434 raft_consensus.cc:2229] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:20.064159 24434 raft_consensus.cc:2258] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:20.064386 24435 ts_tablet_manager.cc:1905] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:20.066309 24435 ts_tablet_manager.cc:1918] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:27:20.066322 24434 ts_tablet_manager.cc:1905] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:20.066378 24435 log.cc:1196] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/9ef2e81709314e62bcd6fca7fa97b70d
I20260619 20:27:20.066453 24433 tablet_replica.cc:330] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b: stopping tablet replica
I20260619 20:27:20.066520 24433 raft_consensus.cc:2229] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:20.066581 24433 raft_consensus.cc:2258] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:20.066766 24433 ts_tablet_manager.cc:1905] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:20.066792 24435 ts_tablet_manager.cc:1939] T 9ef2e81709314e62bcd6fca7fa97b70d P 66c4a24fb82a4f508f924b0878824024: Deleting consensus metadata
I20260619 20:27:20.067116 23836 catalog_manager.cc:4841] TS 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215): tablet 9ef2e81709314e62bcd6fca7fa97b70d (table test-table [id=8da3082782b2405ead5e3c4728281994]) successfully deleted
I20260619 20:27:20.067909 24434 ts_tablet_manager.cc:1918] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:27:20.067963 24434 log.cc:1196] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/9ef2e81709314e62bcd6fca7fa97b70d
I20260619 20:27:20.068163 24434 ts_tablet_manager.cc:1939] T 9ef2e81709314e62bcd6fca7fa97b70d P 55f4a0f200a6437a9a734e071b535003: Deleting consensus metadata
I20260619 20:27:20.068559 24433 ts_tablet_manager.cc:1918] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:27:20.068569 23836 catalog_manager.cc:4841] TS 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263): tablet 9ef2e81709314e62bcd6fca7fa97b70d (table test-table [id=8da3082782b2405ead5e3c4728281994]) successfully deleted
I20260619 20:27:20.068604 24433 log.cc:1196] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/9ef2e81709314e62bcd6fca7fa97b70d
I20260619 20:27:20.068818 24433 ts_tablet_manager.cc:1939] T 9ef2e81709314e62bcd6fca7fa97b70d P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting consensus metadata
I20260619 20:27:20.069295 23837 catalog_manager.cc:4841] TS 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985): tablet 9ef2e81709314e62bcd6fca7fa97b70d (table test-table [id=8da3082782b2405ead5e3c4728281994]) successfully deleted
Jun 19 20:27:20 dist-test-slave-f9rk krb5kdc[23806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900840, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:27:20 dist-test-slave-f9rk krb5kdc[23806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900840, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:20.086995 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:20.079980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.21.151.62:35593)
0619 20:27:20.080120 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:20.080122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:20.080131 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:20.080199 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:20.080200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:20.080210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:20.080306 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:20.081069 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.081623 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.082307 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.082493 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.082540 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:20.084785 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:20.084786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:20.084787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:20.084794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:20.086062 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:20.086356 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:20.086357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:20.086357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:20.086406 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:20.086570 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:20.086571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:20.086571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:20.086675 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:20.086751 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:20.086830 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 20:27:20.086890 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":87,"threads_started":1}
I20260619 20:27:20.088657 23851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:20.088795 23851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:20.090119 24128 tablet_service.cc:1467] Processing CreateTablet for tablet ff4c1c884a374794a80ed0f218e8782e (DEFAULT_TABLE table=test-table [id=4c37afab4ebb4477915a80c640b5c9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:20.090126 24268 tablet_service.cc:1467] Processing CreateTablet for tablet ff4c1c884a374794a80ed0f218e8782e (DEFAULT_TABLE table=test-table [id=4c37afab4ebb4477915a80c640b5c9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:20.090116 23988 tablet_service.cc:1467] Processing CreateTablet for tablet ff4c1c884a374794a80ed0f218e8782e (DEFAULT_TABLE table=test-table [id=4c37afab4ebb4477915a80c640b5c9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:20.090252 24128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4c1c884a374794a80ed0f218e8782e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:20.090252 24268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4c1c884a374794a80ed0f218e8782e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:20.090271 23988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4c1c884a374794a80ed0f218e8782e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:20.091401 24444 tablet_bootstrap.cc:492] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003: Bootstrap starting.
I20260619 20:27:20.091566 24445 tablet_bootstrap.cc:492] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024: Bootstrap starting.
I20260619 20:27:20.091720 24446 tablet_bootstrap.cc:492] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b: Bootstrap starting.
I20260619 20:27:20.091897 24444 tablet_bootstrap.cc:654] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:20.092123 24446 tablet_bootstrap.cc:654] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:20.092571 24444 tablet_bootstrap.cc:492] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003: No bootstrap required, opened a new log
I20260619 20:27:20.092630 24444 ts_tablet_manager.cc:1397] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:20.092684 24446 tablet_bootstrap.cc:492] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b: No bootstrap required, opened a new log
I20260619 20:27:20.092701 24445 tablet_bootstrap.cc:654] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:20.092736 24446 ts_tablet_manager.cc:1397] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:20.092749 24444 raft_consensus.cc:348] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.092837 24444 raft_consensus.cc:374] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:20.092867 24444 raft_consensus.cc:729] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f4a0f200a6437a9a734e071b535003, State: Initialized, Role: FOLLOWER
I20260619 20:27:20.092852 24446 raft_consensus.cc:348] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.092931 24446 raft_consensus.cc:374] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:20.092963 24446 raft_consensus.cc:729] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f028faf9bbc49c79ceaa0e92ef2859b, State: Initialized, Role: FOLLOWER
I20260619 20:27:20.092931 24444 consensus_queue.cc:260] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.093011 24446 consensus_queue.cc:260] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.093042 24444 ts_tablet_manager.cc:1428] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:20.093120 24446 ts_tablet_manager.cc:1428] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:20.093592 24445 tablet_bootstrap.cc:492] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024: No bootstrap required, opened a new log
I20260619 20:27:20.093644 24445 ts_tablet_manager.cc:1397] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:20.093780 24445 raft_consensus.cc:348] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.093851 24445 raft_consensus.cc:374] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:20.093865 24445 raft_consensus.cc:729] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c4a24fb82a4f508f924b0878824024, State: Initialized, Role: FOLLOWER
I20260619 20:27:20.093941 24445 consensus_queue.cc:260] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.094118 24445 ts_tablet_manager.cc:1428] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:20.167326 24411 raft_consensus.cc:482] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:20.167441 24411 raft_consensus.cc:504] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.167697 24411 leader_election.cc:288] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215), 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985)
I20260619 20:27:20.167868 24147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4c1c884a374794a80ed0f218e8782e" candidate_uuid: "55f4a0f200a6437a9a734e071b535003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c4a24fb82a4f508f924b0878824024" is_pre_election: true
I20260619 20:27:20.167855 24288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4c1c884a374794a80ed0f218e8782e" candidate_uuid: "55f4a0f200a6437a9a734e071b535003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" is_pre_election: true
I20260619 20:27:20.167961 24288 raft_consensus.cc:2454] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55f4a0f200a6437a9a734e071b535003 in term 0.
I20260619 20:27:20.167961 24147 raft_consensus.cc:2454] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55f4a0f200a6437a9a734e071b535003 in term 0.
I20260619 20:27:20.168108 23941 leader_election.cc:302] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f4a0f200a6437a9a734e071b535003, 7f028faf9bbc49c79ceaa0e92ef2859b; no voters:
I20260619 20:27:20.168222 24411 raft_consensus.cc:2781] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:20.168284 24411 raft_consensus.cc:482] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:20.168313 24411 raft_consensus.cc:3037] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:20.169067 24411 raft_consensus.cc:504] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.169210 24411 leader_election.cc:288] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [CANDIDATE]: Term 1 election: Requested vote from peers 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215), 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985)
I20260619 20:27:20.169334 24288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4c1c884a374794a80ed0f218e8782e" candidate_uuid: "55f4a0f200a6437a9a734e071b535003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b"
I20260619 20:27:20.169353 24147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4c1c884a374794a80ed0f218e8782e" candidate_uuid: "55f4a0f200a6437a9a734e071b535003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c4a24fb82a4f508f924b0878824024"
I20260619 20:27:20.169414 24288 raft_consensus.cc:3037] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:20.169435 24147 raft_consensus.cc:3037] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:20.170006 24288 raft_consensus.cc:2454] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55f4a0f200a6437a9a734e071b535003 in term 1.
I20260619 20:27:20.170149 24147 raft_consensus.cc:2454] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55f4a0f200a6437a9a734e071b535003 in term 1.
I20260619 20:27:20.170255 23941 leader_election.cc:302] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f4a0f200a6437a9a734e071b535003, 7f028faf9bbc49c79ceaa0e92ef2859b; no voters:
I20260619 20:27:20.170346 24411 raft_consensus.cc:2781] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:20.170409 24411 raft_consensus.cc:686] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 1 LEADER]: Becoming Leader. State: Replica: 55f4a0f200a6437a9a734e071b535003, State: Running, Role: LEADER
I20260619 20:27:20.170462 24411 consensus_queue.cc:237] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.171063 23851 catalog_manager.cc:5495] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55f4a0f200a6437a9a734e071b535003 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "55f4a0f200a6437a9a734e071b535003" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } health_report { overall_health: UNKNOWN } } }
I20260619 20:27:20.186767 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:20.184794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.21.151.1:41263)
0619 20:27:20.184926 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:20.184928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:20.184935 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:27:20.184958 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:20.184963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:20.184973 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:20.185027 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:20.185119 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.185589 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.186264 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.186439 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.186495 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:20.186595 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:20.186620 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:20.186657 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:20.186696 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 20:27:20.186712 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":89,"threads_started":1}
I20260619 20:27:20.190692 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:20.188792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:57995 (local address 127.21.151.1:41263)
0619 20:27:20.188825 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:20.188826 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:20.188925 (+ 99us) server_negotiation.cc:407] Connection header received
0619 20:27:20.188981 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:20.188982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:20.188990 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:20.189024 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:20.189114 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.189544 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.190195 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.190364 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.190412 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:20.190507 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:20.190527 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:20.190559 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:20.190608 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 20:27:20.190621 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:27:20.190711 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:20.188743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:41263 (local address 127.21.151.1:57995)
0619 20:27:20.188871 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:20.188880 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:20.188934 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:20.189041 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:20.189042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:20.189043 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:20.189076 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:20.189077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.189597 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.189597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:20.190125 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:20.190127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.190424 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.190425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:20.190461 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:20.190464 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:20.190571 (+ 107us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:20.190572 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:27:20.190601 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 20:27:20.190631 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":95,"threads_started":1}
I20260619 20:27:20.192476 24147 raft_consensus.cc:1261] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Refusing update from remote peer 55f4a0f200a6437a9a734e071b535003: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:20.192492 24288 raft_consensus.cc:1261] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Refusing update from remote peer 55f4a0f200a6437a9a734e071b535003: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:20.192692 24419 consensus_queue.cc:1035] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:20.192790 24414 consensus_queue.cc:1035] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:20.202484 23850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:20.202579 23850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:20.203220 23850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94: Sending DeleteTablet for 3 replicas of tablet ff4c1c884a374794a80ed0f218e8782e
I20260619 20:27:20.203469 24268 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4c1c884a374794a80ed0f218e8782e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:20 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:58044
I20260619 20:27:20.203557 24433 tablet_replica.cc:330] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b: stopping tablet replica
I20260619 20:27:20.203585 24128 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4c1c884a374794a80ed0f218e8782e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:20 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:44730
I20260619 20:27:20.203594 23988 tablet_service.cc:1514] Processing DeleteTablet for tablet ff4c1c884a374794a80ed0f218e8782e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:20 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57224
I20260619 20:27:20.203603 24433 raft_consensus.cc:2229] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:20.203670 24433 raft_consensus.cc:2258] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:20.203689 24434 tablet_replica.cc:330] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003: stopping tablet replica
I20260619 20:27:20.203699 24435 tablet_replica.cc:330] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024: stopping tablet replica
I20260619 20:27:20.203734 24434 raft_consensus.cc:2229] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:20.203755 24435 raft_consensus.cc:2229] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:20.203806 24434 raft_consensus.cc:2258] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:20.203810 24435 raft_consensus.cc:2258] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:20.204005 24433 ts_tablet_manager.cc:1905] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:20.204070 24434 ts_tablet_manager.cc:1905] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:20.204310 23850 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:20.204437 23850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:20.205662 24434 ts_tablet_manager.cc:1918] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:20.205724 24434 log.cc:1196] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/ff4c1c884a374794a80ed0f218e8782e
I20260619 20:27:20.205744 24435 ts_tablet_manager.cc:1905] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:20.205811 24128 tablet_service.cc:1467] Processing CreateTablet for tablet 18a3483eb8f04ea19a4f47119734764b (DEFAULT_TABLE table=test-table [id=dd4b289656a242858988b4ab4cda0091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:20.205914 24128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a3483eb8f04ea19a4f47119734764b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:20.205973 24434 ts_tablet_manager.cc:1939] T ff4c1c884a374794a80ed0f218e8782e P 55f4a0f200a6437a9a734e071b535003: Deleting consensus metadata
I20260619 20:27:20.206035 23988 tablet_service.cc:1467] Processing CreateTablet for tablet 18a3483eb8f04ea19a4f47119734764b (DEFAULT_TABLE table=test-table [id=dd4b289656a242858988b4ab4cda0091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:20.206116 23988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a3483eb8f04ea19a4f47119734764b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:20.207051 23836 catalog_manager.cc:4841] TS 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263): tablet ff4c1c884a374794a80ed0f218e8782e (table test-table [id=4c37afab4ebb4477915a80c640b5c9e8]) successfully deleted
I20260619 20:27:20.207101 24444 tablet_bootstrap.cc:492] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003: Bootstrap starting.
I20260619 20:27:20.207125 24435 ts_tablet_manager.cc:1918] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:20.207167 24435 log.cc:1196] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/ff4c1c884a374794a80ed0f218e8782e
I20260619 20:27:20.207401 24435 ts_tablet_manager.cc:1939] T ff4c1c884a374794a80ed0f218e8782e P 66c4a24fb82a4f508f924b0878824024: Deleting consensus metadata
I20260619 20:27:20.207428 24444 tablet_bootstrap.cc:654] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:20.207554 24268 tablet_service.cc:1467] Processing CreateTablet for tablet 18a3483eb8f04ea19a4f47119734764b (DEFAULT_TABLE table=test-table [id=dd4b289656a242858988b4ab4cda0091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:20.207669 24445 tablet_bootstrap.cc:492] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024: Bootstrap starting.
I20260619 20:27:20.207762 23836 catalog_manager.cc:4841] TS 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215): tablet ff4c1c884a374794a80ed0f218e8782e (table test-table [id=4c37afab4ebb4477915a80c640b5c9e8]) successfully deleted
I20260619 20:27:20.208057 24444 tablet_bootstrap.cc:492] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003: No bootstrap required, opened a new log
I20260619 20:27:20.208070 24445 tablet_bootstrap.cc:654] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:20.208122 24444 ts_tablet_manager.cc:1397] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 20:27:20.208230 24444 raft_consensus.cc:348] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.208305 24444 raft_consensus.cc:374] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:20.208329 24444 raft_consensus.cc:729] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f4a0f200a6437a9a734e071b535003, State: Initialized, Role: FOLLOWER
I20260619 20:27:20.208376 24444 consensus_queue.cc:260] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.208436 24433 ts_tablet_manager.cc:1918] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:20.208482 24444 ts_tablet_manager.cc:1428] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:20.208500 24433 log.cc:1196] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/ff4c1c884a374794a80ed0f218e8782e
I20260619 20:27:20.208700 24433 ts_tablet_manager.cc:1939] T ff4c1c884a374794a80ed0f218e8782e P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting consensus metadata
I20260619 20:27:20.208441 24268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a3483eb8f04ea19a4f47119734764b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:20.208949 23837 catalog_manager.cc:4841] TS 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985): tablet ff4c1c884a374794a80ed0f218e8782e (table test-table [id=4c37afab4ebb4477915a80c640b5c9e8]) successfully deleted
I20260619 20:27:20.209820 24446 tablet_bootstrap.cc:492] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b: Bootstrap starting.
I20260619 20:27:20.209884 24445 tablet_bootstrap.cc:492] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024: No bootstrap required, opened a new log
I20260619 20:27:20.209959 24445 ts_tablet_manager.cc:1397] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:20.210099 24445 raft_consensus.cc:348] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.210153 24445 raft_consensus.cc:374] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:20.210166 24445 raft_consensus.cc:729] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c4a24fb82a4f508f924b0878824024, State: Initialized, Role: FOLLOWER
I20260619 20:27:20.210171 24446 tablet_bootstrap.cc:654] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:20.210222 24445 consensus_queue.cc:260] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.210314 24445 ts_tablet_manager.cc:1428] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:20.212005 24446 tablet_bootstrap.cc:492] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b: No bootstrap required, opened a new log
I20260619 20:27:20.212057 24446 ts_tablet_manager.cc:1397] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:20.212162 24446 raft_consensus.cc:348] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.212227 24446 raft_consensus.cc:374] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:20.212247 24446 raft_consensus.cc:729] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f028faf9bbc49c79ceaa0e92ef2859b, State: Initialized, Role: FOLLOWER
I20260619 20:27:20.212288 24446 consensus_queue.cc:260] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.212378 24446 ts_tablet_manager.cc:1428] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:20.489248 24450 raft_consensus.cc:482] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:20.489370 24450 raft_consensus.cc:504] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.489559 24450 leader_election.cc:288] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263), 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985)
I20260619 20:27:20.489732 24008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a3483eb8f04ea19a4f47119734764b" candidate_uuid: "66c4a24fb82a4f508f924b0878824024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f4a0f200a6437a9a734e071b535003" is_pre_election: true
I20260619 20:27:20.489842 24008 raft_consensus.cc:2454] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c4a24fb82a4f508f924b0878824024 in term 0.
I20260619 20:27:20.489773 24288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a3483eb8f04ea19a4f47119734764b" candidate_uuid: "66c4a24fb82a4f508f924b0878824024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" is_pre_election: true
I20260619 20:27:20.489883 24288 raft_consensus.cc:2454] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66c4a24fb82a4f508f924b0878824024 in term 0.
I20260619 20:27:20.490002 24080 leader_election.cc:302] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55f4a0f200a6437a9a734e071b535003, 66c4a24fb82a4f508f924b0878824024; no voters:
I20260619 20:27:20.490125 24450 raft_consensus.cc:2781] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:20.490168 24450 raft_consensus.cc:482] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:20.490201 24450 raft_consensus.cc:3037] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:20.490883 24450 raft_consensus.cc:504] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.491029 24450 leader_election.cc:288] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [CANDIDATE]: Term 1 election: Requested vote from peers 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263), 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985)
I20260619 20:27:20.491156 24008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a3483eb8f04ea19a4f47119734764b" candidate_uuid: "66c4a24fb82a4f508f924b0878824024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f4a0f200a6437a9a734e071b535003"
I20260619 20:27:20.491219 24288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a3483eb8f04ea19a4f47119734764b" candidate_uuid: "66c4a24fb82a4f508f924b0878824024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b"
I20260619 20:27:20.491263 24008 raft_consensus.cc:3037] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:20.491281 24288 raft_consensus.cc:3037] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:20.492064 24008 raft_consensus.cc:2454] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c4a24fb82a4f508f924b0878824024 in term 1.
I20260619 20:27:20.492064 24288 raft_consensus.cc:2454] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c4a24fb82a4f508f924b0878824024 in term 1.
I20260619 20:27:20.492230 24081 leader_election.cc:302] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66c4a24fb82a4f508f924b0878824024, 7f028faf9bbc49c79ceaa0e92ef2859b; no voters:
I20260619 20:27:20.492364 24450 raft_consensus.cc:2781] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:20.492511 24450 raft_consensus.cc:686] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 1 LEADER]: Becoming Leader. State: Replica: 66c4a24fb82a4f508f924b0878824024, State: Running, Role: LEADER
I20260619 20:27:20.492568 24450 consensus_queue.cc:237] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } }
I20260619 20:27:20.493294 23850 catalog_manager.cc:5495] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66c4a24fb82a4f508f924b0878824024 (127.21.151.2). New cstate: current_term: 1 leader_uuid: "66c4a24fb82a4f508f924b0878824024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66c4a24fb82a4f508f924b0878824024" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 } health_report { overall_health: UNKNOWN } } }
I20260619 20:27:20.565202 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:20.562912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.21.151.2:35215)
0619 20:27:20.563071 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:20.563072 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:20.563079 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:27:20.563116 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:20.563117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:20.563136 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:20.563205 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:20.563298 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.563893 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.564536 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.564726 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.564773 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:20.564886 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:20.564985 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:20.565034 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:20.565067 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 20:27:20.565082 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":112,"threads_started":1}
I20260619 20:27:20.569775 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:20.567723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:60841 (local address 127.21.151.1:41263)
0619 20:27:20.567755 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:20.567757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:20.567811 (+ 54us) server_negotiation.cc:407] Connection header received
0619 20:27:20.567897 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:20.567898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:20.567907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:20.567948 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:20.568097 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.568515 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.569211 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.569380 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.569427 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:20.569566 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:20.569615 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:20.569652 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:20.569708 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 20:27:20.569723 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260619 20:27:20.569810 24468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:20.567556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:41263 (local address 127.21.151.2:60841)
0619 20:27:20.567752 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:20.567763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:20.567828 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:20.567968 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:20.567969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:20.567970 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:20.568041 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:20.568042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.568579 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.568580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:20.569120 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:20.569121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.569444 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.569445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:20.569490 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:20.569501 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:20.569671 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:20.569673 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:27:20.569707 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 20:27:20.569741 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":112,"threads_started":1}
I20260619 20:27:20.571695 24008 raft_consensus.cc:1261] T 18a3483eb8f04ea19a4f47119734764b P 55f4a0f200a6437a9a734e071b535003 [term 1 FOLLOWER]: Refusing update from remote peer 66c4a24fb82a4f508f924b0878824024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:20.571889 24288 raft_consensus.cc:1261] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Refusing update from remote peer 66c4a24fb82a4f508f924b0878824024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:20.571906 24450 consensus_queue.cc:1035] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55f4a0f200a6437a9a734e071b535003" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 41263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:20.572184 24450 consensus_queue.cc:1035] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f028faf9bbc49c79ceaa0e92ef2859b" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:20.578685 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:20.576355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:58327 (local address 127.21.151.2:35215)
0619 20:27:20.576397 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:20.576399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:20.576405 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:27:20.576421 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:20.576421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:20.576429 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:20.576487 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:20.576602 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.577234 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.577907 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.578162 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.578236 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:20.578386 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:20.578424 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:20.578466 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:20.578582 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 20:27:20.578603 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:27:20.578699 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:20.576286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:35215 (local address 127.21.151.1:58327)
0619 20:27:20.576321 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:20.576332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:20.576371 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:20.576505 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:20.576506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:20.576507 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:20.576540 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:20.576542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.577311 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.577312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:20.577827 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:20.577828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:20.578235 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:20.578235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:20.578289 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:20.578319 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:20.578515 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:20.578517 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:27:20.578564 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 20:27:20.578613 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 20:27:20.579128 24415 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 20:27:22.097440 23849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:22.097537 23849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:22.098292 23849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b7a22643b794a859b0ae0ab15ef8d94: Sending DeleteTablet for 3 replicas of tablet 18a3483eb8f04ea19a4f47119734764b
I20260619 20:27:22.098526 23988 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a3483eb8f04ea19a4f47119734764b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:22 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57224
I20260619 20:27:22.098558 24128 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a3483eb8f04ea19a4f47119734764b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:22 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:44730
I20260619 20:27:22.098670 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 23923
I20260619 20:27:22.098696 24268 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a3483eb8f04ea19a4f47119734764b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:22 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:58044
I20260619 20:27:22.099462 24495 tablet_replica.cc:330] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024: stopping tablet replica
I20260619 20:27:22.099520 24495 raft_consensus.cc:2229] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:22.099568 24495 raft_consensus.cc:2258] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:22.099856 24495 ts_tablet_manager.cc:1905] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:22.101040 24496 tablet_replica.cc:330] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b: stopping tablet replica
I20260619 20:27:22.101125 24496 raft_consensus.cc:2229] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:22.101172 24496 raft_consensus.cc:2258] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:22.101434 24495 ts_tablet_manager.cc:1918] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:27:22.101470 24496 ts_tablet_manager.cc:1905] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:22.101509 24495 log.cc:1196] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/18a3483eb8f04ea19a4f47119734764b
I20260619 20:27:22.101768 24495 ts_tablet_manager.cc:1939] T 18a3483eb8f04ea19a4f47119734764b P 66c4a24fb82a4f508f924b0878824024: Deleting consensus metadata
I20260619 20:27:22.102059 23836 catalog_manager.cc:4841] TS 66c4a24fb82a4f508f924b0878824024 (127.21.151.2:35215): tablet 18a3483eb8f04ea19a4f47119734764b (table test-table [id=dd4b289656a242858988b4ab4cda0091]) successfully deleted
I20260619 20:27:22.103322 24496 ts_tablet_manager.cc:1918] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 20:27:22.103396 24496 log.cc:1196] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/18a3483eb8f04ea19a4f47119734764b
I20260619 20:27:22.103606 24496 ts_tablet_manager.cc:1939] T 18a3483eb8f04ea19a4f47119734764b P 7f028faf9bbc49c79ceaa0e92ef2859b: Deleting consensus metadata
I20260619 20:27:22.104027 23837 catalog_manager.cc:4841] TS 7f028faf9bbc49c79ceaa0e92ef2859b (127.21.151.3:43985): tablet 18a3483eb8f04ea19a4f47119734764b (table test-table [id=dd4b289656a242858988b4ab4cda0091]) successfully deleted
W20260619 20:27:22.106205 23836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:41263 (error 108)
I20260619 20:27:22.106452 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 24063
I20260619 20:27:22.107225 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:22.106557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:41263 (local address 127.0.0.1:52050)
0619 20:27:22.107129 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:22.107178 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:41263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":55,"threads_started":1}
W20260619 20:27:22.107326 23836 catalog_manager.cc:4584] TS 55f4a0f200a6437a9a734e071b535003 (127.21.151.1:41263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18a3483eb8f04ea19a4f47119734764b: Network error: Client connection negotiation failed: client connection to 127.21.151.1:41263: connect: Connection refused (error 111)
I20260619 20:27:22.112442 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 24203
I20260619 20:27:22.118000 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 23819
2026-06-19T20:27:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5991 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:22 dist-test-slave-f9rk krb5kdc[24501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:22 dist-test-slave-f9rk krb5kdc[24501](info): set up 2 sockets
Jun 19 20:27:22 dist-test-slave-f9rk krb5kdc[24501](info): commencing operation
krb5kdc: starting...
W20260619 20:27:23.188436 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:23 dist-test-slave-f9rk krb5kdc[24501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:23.302431 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:42203
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:42203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:23.382817 24514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:23.383044 24514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:23.383090 24514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:23.384268 24514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:23.384336 24514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:23.384359 24514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:23.384379 24514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:23.384408 24514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:23.385851 24514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:42203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:42203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:23.386143 24514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:23.386379 24514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:23.391408 24520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:23.391428 24519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:23.391450 24522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:23.391607 24514 server_base.cc:1034] running on GCE node
I20260619 20:27:23.391870 24514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:23.392160 24514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:23.393335 24514 hybrid_clock.cc:648] HybridClock initialized: now 1781900843393319 us; error 28 us; skew 500 ppm
Jun 19 20:27:23 dist-test-slave-f9rk krb5kdc[24501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:23.398641 24514 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:23.399107 24514 webserver.cc:460] Webserver started at http://127.21.151.62:45043/ using document root <none> and password file <none>
I20260619 20:27:23.399345 24514 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:23.399415 24514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:23.399565 24514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:23.400643 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "7dd3838eb5a542beb9c08bd37df30033"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "6f9a2dd17fdd451268307c90284ea819"
server_key_iv: "22d0cce58050ecca4fa8a1f7016d5408"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.401008 24514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "7dd3838eb5a542beb9c08bd37df30033"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "6f9a2dd17fdd451268307c90284ea819"
server_key_iv: "22d0cce58050ecca4fa8a1f7016d5408"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.402287 24514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:23.402969 24529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:23.403160 24514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:23.403239 24514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "7dd3838eb5a542beb9c08bd37df30033"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "6f9a2dd17fdd451268307c90284ea819"
server_key_iv: "22d0cce58050ecca4fa8a1f7016d5408"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.403290 24514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:23.419319 24514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:23.421657 24514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:23.421765 24514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:23.425532 24514 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:42203
I20260619 20:27:23.425566 24581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:42203 every 8 connection(s)
I20260619 20:27:23.425916 24514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:23.426801 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:23.427132 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 24514
I20260619 20:27:23.427245 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:23.427446 22108 external_mini_cluster.cc:1428] Setting key 45b007fb55f76f38421a56ba02648233
I20260619 20:27:23.429880 24582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033: Bootstrap starting.
I20260619 20:27:23.430541 24582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:23.430845 24582 log.cc:824] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:23.431550 24582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033: No bootstrap required, opened a new log
I20260619 20:27:23.433674 24582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd3838eb5a542beb9c08bd37df30033" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42203 } }
I20260619 20:27:23.433806 24582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:23.433828 24582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dd3838eb5a542beb9c08bd37df30033, State: Initialized, Role: FOLLOWER
I20260619 20:27:23.433957 24582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd3838eb5a542beb9c08bd37df30033" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42203 } }
I20260619 20:27:23.434041 24582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:23.434099 24582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:23.434147 24582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 20:27:23 dist-test-slave-f9rk krb5kdc[24501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:23.434716 24582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd3838eb5a542beb9c08bd37df30033" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42203 } }
I20260619 20:27:23.434803 24582 leader_election.cc:302] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dd3838eb5a542beb9c08bd37df30033; no voters:
I20260619 20:27:23.435015 24582 leader_election.cc:288] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:23.435045 24587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:23.435171 24587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [term 1 LEADER]: Becoming Leader. State: Replica: 7dd3838eb5a542beb9c08bd37df30033, State: Running, Role: LEADER
I20260619 20:27:23.435279 24582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:23.435314 24587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd3838eb5a542beb9c08bd37df30033" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42203 } }
I20260619 20:27:23.435671 24587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dd3838eb5a542beb9c08bd37df30033. Latest consensus state: current_term: 1 leader_uuid: "7dd3838eb5a542beb9c08bd37df30033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd3838eb5a542beb9c08bd37df30033" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42203 } } }
I20260619 20:27:23.435770 24587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:23.435863 24589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dd3838eb5a542beb9c08bd37df30033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd3838eb5a542beb9c08bd37df30033" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42203 } } }
I20260619 20:27:23.436035 24589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:23.436283 24596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:23.436810 24596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:23.438548 24596 catalog_manager.cc:1347] Generated new cluster ID: bcae10476f07419297da81233cf8a080
I20260619 20:27:23.438593 24596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:23.439909 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:23.428099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.21.151.62:42203)
0619 20:27:23.428266 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:23.428276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:23.428308 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:27:23.428842 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:23.428850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:23.428911 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:23.429061 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:23.429658 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.431012 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.431940 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.432546 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.432662 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:23.435022 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:23.435026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:23.435027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:23.435040 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:23.438320 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:23.438694 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:23.438697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:23.438697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:23.438792 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:23.439002 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:23.439003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:23.439004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:23.439270 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:23.439411 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:23.439528 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 20:27:23.439698 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260619 20:27:23.457037 24596 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:23.457545 24596 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:23.462818 24596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dd3838eb5a542beb9c08bd37df30033: Generated new TSK 0
I20260619 20:27:23.462998 24596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:27:23.491757 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:42203
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:23.590482 24608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:23.590680 24608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:23.590703 24608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:23.591893 24608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:23.591931 24608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:23.591956 24608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:23.593336 24608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:42203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:23.593578 24608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:23.593801 24608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:23.599364 24613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:23.599363 24616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:23.599401 24614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:23.599704 24608 server_base.cc:1034] running on GCE node
I20260619 20:27:23.599951 24608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:23.600196 24608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:23.601333 24608 hybrid_clock.cc:648] HybridClock initialized: now 1781900843601313 us; error 37 us; skew 500 ppm
Jun 19 20:27:23 dist-test-slave-f9rk krb5kdc[24501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:23.603852 24608 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:27:23.604310 24608 webserver.cc:460] Webserver started at http://127.21.151.1:39975/ using document root <none> and password file <none>
I20260619 20:27:23.604534 24608 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:23.604602 24608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:23.604735 24608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:23.605736 24608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "a4402a33ab094ce4a9474289c600a4c6"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "0cc11e571f9267cfed01fd621f73873e"
server_key_iv: "6478bc945b58d3b5b62dfd71d8a04761"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.606108 24608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "a4402a33ab094ce4a9474289c600a4c6"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "0cc11e571f9267cfed01fd621f73873e"
server_key_iv: "6478bc945b58d3b5b62dfd71d8a04761"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.607509 24608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:23.608253 24623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:23.608420 24608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:23.608515 24608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "a4402a33ab094ce4a9474289c600a4c6"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "0cc11e571f9267cfed01fd621f73873e"
server_key_iv: "6478bc945b58d3b5b62dfd71d8a04761"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.608596 24608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:23.631771 24608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:23.633620 24608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:23.633801 24608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:23.634045 24608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:23.634398 24608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:23.634454 24608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:23.634490 24608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:23.634521 24608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:23.640156 24608 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:43521
I20260619 20:27:23.640267 24736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:43521 every 8 connection(s)
I20260619 20:27:23.640553 24608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:27:23 dist-test-slave-f9rk krb5kdc[24501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:23.647740 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 24608
I20260619 20:27:23.647810 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:23.647981 22108 external_mini_cluster.cc:1428] Setting key 26eb347d35b84de5c72bd7483559ad14
I20260619 20:27:23.649556 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:23.641291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:34675 (local address 127.21.151.62:42203)
0619 20:27:23.641360 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:23.641362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:23.641540 (+ 178us) server_negotiation.cc:407] Connection header received
0619 20:27:23.642030 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:23.642030 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:23.642039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:23.642088 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:23.642991 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.643474 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.644437 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.644627 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.644679 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:23.646651 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:23.646651 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:23.646652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:23.646659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:23.648140 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:23.648586 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:23.648587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:23.648587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:23.648653 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:23.648908 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:23.648909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:23.648910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:23.649024 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:23.649099 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:23.649371 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 20:27:23.649454 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:27:23.649748 24739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:23.641259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:42203 (local address 127.21.151.1:34675)
0619 20:27:23.641416 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:23.641457 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:23.641945 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:23.642149 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:23.642152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:23.642712 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:23.642911 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:23.642925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.643549 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.643549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:23.644366 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:23.644369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.644700 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.644700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:23.644817 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:23.645232 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:23.645239 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:23.646564 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:23.648210 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:23.648216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:23.648218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:23.648502 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:23.648730 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:23.648731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:23.648731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:23.648841 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:23.649090 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:23.649096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:23.649229 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 20:27:23.649333 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 20:27:23.649500 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260619 20:27:23.650391 24737 heartbeater.cc:347] Connected to a master server at 127.21.151.62:42203
I20260619 20:27:23.650488 24737 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:23.650666 24737 heartbeater.cc:511] Master 127.21.151.62:42203 requested a full tablet report, sending...
I20260619 20:27:23.651196 24546 ts_manager.cc:194] Registered new tserver with Master: a4402a33ab094ce4a9474289c600a4c6 (127.21.151.1:43521)
I20260619 20:27:23.652190 24546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:34675
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:27:23.671859 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:42203
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:23.750161 24742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:23.750384 24742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:23.750419 24742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:23.751538 24742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:23.751600 24742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:23.751654 24742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:23.753055 24742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:42203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:23.753321 24742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:23.753579 24742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:23.758404 24748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:23.758471 24742 server_base.cc:1034] running on GCE node
W20260619 20:27:23.758368 24747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:23.758383 24750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:23.758749 24742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:23.758932 24742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:23.760052 24742 hybrid_clock.cc:648] HybridClock initialized: now 1781900843760022 us; error 46 us; skew 500 ppm
Jun 19 20:27:23 dist-test-slave-f9rk krb5kdc[24501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:23.762472 24742 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:27:23.762885 24742 webserver.cc:460] Webserver started at http://127.21.151.2:39689/ using document root <none> and password file <none>
I20260619 20:27:23.763123 24742 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:23.763167 24742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:23.763327 24742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:23.764420 24742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "afebc704bbb444d4ae329a91a77bfce4"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "4c05f98b44eabc8d51da247d1be7ec95"
server_key_iv: "511608490a552d2f11b9c2b788e0b6c6"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.764771 24742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "afebc704bbb444d4ae329a91a77bfce4"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "4c05f98b44eabc8d51da247d1be7ec95"
server_key_iv: "511608490a552d2f11b9c2b788e0b6c6"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.766111 24742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:27:23.766841 24757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:23.767009 24742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:23.767104 24742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "afebc704bbb444d4ae329a91a77bfce4"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "4c05f98b44eabc8d51da247d1be7ec95"
server_key_iv: "511608490a552d2f11b9c2b788e0b6c6"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.767184 24742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:23.779588 24742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:23.781481 24742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:23.781652 24742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:23.781872 24742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:23.782178 24742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:23.782232 24742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:23.782271 24742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:23.782299 24742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:23.788134 24742 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:42737
I20260619 20:27:23.788177 24870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:42737 every 8 connection(s)
I20260619 20:27:23.788504 24742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:27:23 dist-test-slave-f9rk krb5kdc[24501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:23.796039 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 24742
I20260619 20:27:23.796140 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:23.796337 22108 external_mini_cluster.cc:1428] Setting key 662fd3a16ec096a77bf00e5731cdc6bf
I20260619 20:27:23.798619 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:23.789135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:57987 (local address 127.21.151.62:42203)
0619 20:27:23.789198 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:23.789200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:23.789431 (+ 231us) server_negotiation.cc:407] Connection header received
0619 20:27:23.789908 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:23.789909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:23.789918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:23.789986 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:23.790839 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.791336 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.792305 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.792491 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.792545 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:23.794469 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:23.794470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:23.794471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:23.794477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:23.795950 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:23.796468 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:23.796469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:23.796470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:23.796540 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:23.796795 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:23.796796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:23.796796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:23.796907 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:23.796988 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:23.798453 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0619 20:27:23.798535 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:27:23.798789 24873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:23.789127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:42203 (local address 127.21.151.2:57987)
0619 20:27:23.789288 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:23.789328 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:23.789792 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:23.790040 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:23.790043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:23.790583 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:23.790764 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:23.790768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.791410 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.791410 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:23.792226 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:23.792228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.792562 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.792563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:23.792655 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:23.793058 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:23.793065 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:23.794379 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:23.796046 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:23.796053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:23.796055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:23.796359 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:23.796624 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:23.796625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:23.796626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:23.796726 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:23.798201 (+ 1475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:23.798206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:23.798312 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 20:27:23.798429 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 20:27:23.798574 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260619 20:27:23.799259 24871 heartbeater.cc:347] Connected to a master server at 127.21.151.62:42203
I20260619 20:27:23.799361 24871 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:23.799521 24871 heartbeater.cc:511] Master 127.21.151.62:42203 requested a full tablet report, sending...
I20260619 20:27:23.799942 24546 ts_manager.cc:194] Registered new tserver with Master: afebc704bbb444d4ae329a91a77bfce4 (127.21.151.2:42737)
I20260619 20:27:23.800328 24546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:57987
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:27:23.819414 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:42203
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:23.901793 24876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:23.902000 24876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:23.902031 24876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:23.903172 24876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:23.903209 24876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:23.903234 24876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:23.904623 24876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:42203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:23.904798 24876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:23.904966 24876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:23.909736 24884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:23.909780 24882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:23.909747 24881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:23.910029 24876 server_base.cc:1034] running on GCE node
I20260619 20:27:23.910252 24876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:23.910477 24876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:23.911655 24876 hybrid_clock.cc:648] HybridClock initialized: now 1781900843911649 us; error 36 us; skew 500 ppm
Jun 19 20:27:23 dist-test-slave-f9rk krb5kdc[24501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:23.914063 24876 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:27:23.914471 24876 webserver.cc:460] Webserver started at http://127.21.151.3:35325/ using document root <none> and password file <none>
I20260619 20:27:23.914683 24876 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:23.914747 24876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:23.914897 24876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:23.915935 24876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "dd330280e7d245bfaec79213646782b3"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "cac7de4538b21ac844dc2314b8891f77"
server_key_iv: "06f37121113df25ab7db59a53c9b2a09"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.916275 24876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "dd330280e7d245bfaec79213646782b3"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "cac7de4538b21ac844dc2314b8891f77"
server_key_iv: "06f37121113df25ab7db59a53c9b2a09"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.917555 24876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:23.918248 24891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:23.918408 24876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:23.918506 24876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "dd330280e7d245bfaec79213646782b3"
format_stamp: "Formatted at 2026-06-19 20:27:23 on dist-test-slave-f9rk"
server_key: "cac7de4538b21ac844dc2314b8891f77"
server_key_iv: "06f37121113df25ab7db59a53c9b2a09"
server_key_version: "encryptionkey@0"
I20260619 20:27:23.918576 24876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:23.932787 24876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:23.935122 24876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:23.935266 24876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:23.935487 24876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:23.935885 24876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:23.935936 24876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:23.935976 24876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:23.935998 24876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:23.941861 24876 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:38699
I20260619 20:27:23.941917 25004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:38699 every 8 connection(s)
I20260619 20:27:23.942220 24876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:23.943599 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 24876
I20260619 20:27:23.943701 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:23.943900 22108 external_mini_cluster.cc:1428] Setting key e0edf46f129830e26ef6093e92a3355d
Jun 19 20:27:23 dist-test-slave-f9rk krb5kdc[24501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:23.951066 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:23.942898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:38961 (local address 127.21.151.62:42203)
0619 20:27:23.942960 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:23.942961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:23.943179 (+ 218us) server_negotiation.cc:407] Connection header received
0619 20:27:23.943777 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:23.943778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:23.943789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:23.943852 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:23.945002 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.945478 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.946410 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.946606 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.946654 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:23.948625 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:23.948626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:23.948626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:23.948633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:23.950024 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:23.950344 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:23.950345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:23.950345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:23.950393 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:23.950587 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:23.950588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:23.950588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:23.950683 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:23.950730 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:23.950946 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 20:27:23.951004 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:27:23.951233 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:23.942876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:42203 (local address 127.21.151.3:38961)
0619 20:27:23.943050 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:23.943090 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:23.943678 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:23.943924 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:23.943928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:23.944743 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:23.944926 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:23.944930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.945554 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.945554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:23.946339 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:23.946340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:23.946673 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:23.946674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:23.946766 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:23.947196 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:23.947217 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:23.948551 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:23.950090 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:23.950094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:23.950095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:23.950285 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:23.950448 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:23.950449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:23.950449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:23.950534 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:23.950735 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:23.950739 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:23.950843 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 20:27:23.950921 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:27:23.951038 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260619 20:27:23.951686 25005 heartbeater.cc:347] Connected to a master server at 127.21.151.62:42203
I20260619 20:27:23.951771 25005 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:23.951901 25005 heartbeater.cc:511] Master 127.21.151.62:42203 requested a full tablet report, sending...
I20260619 20:27:23.952240 24546 ts_manager.cc:194] Registered new tserver with Master: dd330280e7d245bfaec79213646782b3 (127.21.151.3:38699)
I20260619 20:27:23.952601 24546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:38961
I20260619 20:27:23.955380 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 20:27:23.965950 24585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:23.965018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.21.151.62:42203)
0619 20:27:23.965106 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:23.965108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:23.965219 (+ 111us) server_negotiation.cc:407] Connection header received
0619 20:27:23.965310 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:23.965311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:23.965321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:23.965389 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:23.965884 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44226: BlockingRecv error: recv got EOF from 127.0.0.1:44226 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 20:27:23.966837 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 24608
I20260619 20:27:23.971136 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 24742
I20260619 20:27:23.975189 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 24876
I20260619 20:27:23.979084 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 24514
2026-06-19T20:27:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1859 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:24 dist-test-slave-f9rk krb5kdc[25017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:24 dist-test-slave-f9rk krb5kdc[25017](info): set up 2 sockets
Jun 19 20:27:24 dist-test-slave-f9rk krb5kdc[25017](info): commencing operation
krb5kdc: starting...
W20260619 20:27:25.044567 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:25 dist-test-slave-f9rk krb5kdc[25017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:25.162391 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:46579
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:46579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:25.245213 25030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:25.245428 25030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:25.245465 25030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:25.246675 25030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:25.246737 25030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:25.246762 25030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:25.246793 25030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:25.246819 25030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:25.248332 25030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:46579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:46579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:25.248605 25030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:25.248826 25030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:25.253686 25038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:25.253686 25035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:25.253713 25036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:25.253706 25030 server_base.cc:1034] running on GCE node
I20260619 20:27:25.254115 25030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:25.254454 25030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:25.255666 25030 hybrid_clock.cc:648] HybridClock initialized: now 1781900845255652 us; error 47 us; skew 500 ppm
Jun 19 20:27:25 dist-test-slave-f9rk krb5kdc[25017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900845, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:25.258131 25030 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:25.258613 25030 webserver.cc:460] Webserver started at http://127.21.151.62:43675/ using document root <none> and password file <none>
I20260619 20:27:25.258859 25030 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:25.258929 25030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:25.259078 25030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:25.260210 25030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "9fa8295a10e64985956416d3bc7ad497"
format_stamp: "Formatted at 2026-06-19 20:27:25 on dist-test-slave-f9rk"
server_key: "859cfcf078865ddfe4eeee22a4ffe978"
server_key_iv: "ed18ead13da47160ea76004990e5fc04"
server_key_version: "encryptionkey@0"
I20260619 20:27:25.260568 25030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "9fa8295a10e64985956416d3bc7ad497"
format_stamp: "Formatted at 2026-06-19 20:27:25 on dist-test-slave-f9rk"
server_key: "859cfcf078865ddfe4eeee22a4ffe978"
server_key_iv: "ed18ead13da47160ea76004990e5fc04"
server_key_version: "encryptionkey@0"
I20260619 20:27:25.262207 25030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:25.262948 25045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:25.263126 25030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:25.263226 25030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "9fa8295a10e64985956416d3bc7ad497"
format_stamp: "Formatted at 2026-06-19 20:27:25 on dist-test-slave-f9rk"
server_key: "859cfcf078865ddfe4eeee22a4ffe978"
server_key_iv: "ed18ead13da47160ea76004990e5fc04"
server_key_version: "encryptionkey@0"
I20260619 20:27:25.263317 25030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:25.270812 25030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:25.272771 25030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:25.272930 25030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:25.276968 25030 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:46579
I20260619 20:27:25.276999 25097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:46579 every 8 connection(s)
I20260619 20:27:25.277468 25030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:25.278086 25098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:25.280324 25098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497: Bootstrap starting.
I20260619 20:27:25.280903 25098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:25.281201 25098 log.cc:824] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:25.281821 25098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497: No bootstrap required, opened a new log
I20260619 20:27:25.283356 25098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa8295a10e64985956416d3bc7ad497" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46579 } }
I20260619 20:27:25.283488 25098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:25.283510 25098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa8295a10e64985956416d3bc7ad497, State: Initialized, Role: FOLLOWER
I20260619 20:27:25.283703 25098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa8295a10e64985956416d3bc7ad497" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46579 } }
I20260619 20:27:25.283767 25098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:25.283788 25098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:25.283854 25098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:25.284688 25098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa8295a10e64985956416d3bc7ad497" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46579 } }
I20260619 20:27:25.284797 25098 leader_election.cc:302] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa8295a10e64985956416d3bc7ad497; no voters:
I20260619 20:27:25.284984 25098 leader_election.cc:288] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:25.285051 25101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:25.285243 25101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [term 1 LEADER]: Becoming Leader. State: Replica: 9fa8295a10e64985956416d3bc7ad497, State: Running, Role: LEADER
I20260619 20:27:25.285254 25098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:25.285338 25101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa8295a10e64985956416d3bc7ad497" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46579 } }
I20260619 20:27:25.285770 25103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa8295a10e64985956416d3bc7ad497. Latest consensus state: current_term: 1 leader_uuid: "9fa8295a10e64985956416d3bc7ad497" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa8295a10e64985956416d3bc7ad497" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46579 } } }
I20260619 20:27:25.285794 25102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fa8295a10e64985956416d3bc7ad497" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa8295a10e64985956416d3bc7ad497" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46579 } } }
I20260619 20:27:25.286024 25102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:25.286201 25103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:25.286347 25110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:25.286823 25110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:25.287094 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 25030
I20260619 20:27:25.287187 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:25.287324 22108 external_mini_cluster.cc:1428] Setting key afb6d6da52ac77f5cec4c4088ed5c352
I20260619 20:27:25.289083 25110 catalog_manager.cc:1347] Generated new cluster ID: 7a14847107854c5e959e7641762a9239
I20260619 20:27:25.289137 25110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:27:25 dist-test-slave-f9rk krb5kdc[25017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:25.299058 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:25.287730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.21.151.62:46579)
0619 20:27:25.288054 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:25.288064 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:25.288111 (+ 47us) server_negotiation.cc:407] Connection header received
0619 20:27:25.288215 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:25.288220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:25.288261 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:25.288404 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:25.289050 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:25.290431 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:25.291519 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:25.292185 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:25.292277 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:25.295198 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:25.295203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:25.295203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:25.295215 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:25.297215 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:25.297607 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:25.297610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:25.297610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:25.297736 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:25.297952 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:25.297953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:25.297953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:25.298284 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:25.298439 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:25.298524 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:27:25.298771 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":99,"threads_started":1}
I20260619 20:27:25.299927 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:27:25.324257 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:25.319923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.21.151.62:46579)
0619 20:27:25.319965 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:25.319966 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:25.320154 (+ 188us) server_negotiation.cc:407] Connection header received
0619 20:27:25.320225 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:25.320226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:25.320238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:25.320298 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:25.320737 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:25.321504 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:25.322579 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:25.322903 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:25.322964 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:25.323726 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:25.323727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:25.323727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:25.323993 (+ 266us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:25.324014 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:25.324022 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:25.324091 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:25.324150 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 20:27:25.324173 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:25.324920 25110 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:25.325408 25110 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:25.333906 25110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fa8295a10e64985956416d3bc7ad497: Generated new TSK 0
I20260619 20:27:25.334110 25110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:27:25.337051 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 25030
2026-06-19T20:27:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1356 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:25 dist-test-slave-f9rk krb5kdc[25131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:25 dist-test-slave-f9rk krb5kdc[25131](info): set up 2 sockets
Jun 19 20:27:25 dist-test-slave-f9rk krb5kdc[25131](info): commencing operation
krb5kdc: starting...
W20260619 20:27:26.352573 22108 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:26 dist-test-slave-f9rk krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:26.464556 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:34321
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:34321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:26.543049 25144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:26.543246 25144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:26.543267 25144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:26.544445 25144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:26.544484 25144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:26.544498 25144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:26.544512 25144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:26.544562 25144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:26.545971 25144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:34321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:34321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:26.546193 25144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:26.546416 25144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:26.551095 25149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:26.551059 25152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:26.551108 25150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:26.551357 25144 server_base.cc:1034] running on GCE node
I20260619 20:27:26.551596 25144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:26.552011 25144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:26.553161 25144 hybrid_clock.cc:648] HybridClock initialized: now 1781900846553144 us; error 38 us; skew 500 ppm
Jun 19 20:27:26 dist-test-slave-f9rk krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900846, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:26.555706 25144 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:26.556210 25144 webserver.cc:460] Webserver started at http://127.21.151.62:45777/ using document root <none> and password file <none>
I20260619 20:27:26.556435 25144 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:26.556502 25144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:26.556663 25144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:26.557766 25144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "bddd5ccf942e4f49b5c783962bb8f037"
format_stamp: "Formatted at 2026-06-19 20:27:26 on dist-test-slave-f9rk"
server_key: "2210f92f7dd3b161880f9418072847d9"
server_key_iv: "d31ed4af98dde9843df755c977d5b0b8"
server_key_version: "encryptionkey@0"
I20260619 20:27:26.558152 25144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "bddd5ccf942e4f49b5c783962bb8f037"
format_stamp: "Formatted at 2026-06-19 20:27:26 on dist-test-slave-f9rk"
server_key: "2210f92f7dd3b161880f9418072847d9"
server_key_iv: "d31ed4af98dde9843df755c977d5b0b8"
server_key_version: "encryptionkey@0"
I20260619 20:27:26.559574 25144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:27:26.560432 25159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:26.560634 25144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:26.560734 25144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "bddd5ccf942e4f49b5c783962bb8f037"
format_stamp: "Formatted at 2026-06-19 20:27:26 on dist-test-slave-f9rk"
server_key: "2210f92f7dd3b161880f9418072847d9"
server_key_iv: "d31ed4af98dde9843df755c977d5b0b8"
server_key_version: "encryptionkey@0"
I20260619 20:27:26.560822 25144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:26.577750 25144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:26.580205 25144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:26.580377 25144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:26.584573 25144 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:34321
I20260619 20:27:26.584601 25211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:34321 every 8 connection(s)
I20260619 20:27:26.584968 25144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:26.585889 25212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:26.588019 25212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037: Bootstrap starting.
I20260619 20:27:26.588562 25212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:26.588845 25212 log.cc:824] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:26.589128 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 25144
I20260619 20:27:26.589231 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:26.589408 22108 external_mini_cluster.cc:1428] Setting key 083ad30557f99b4ba225be322d026df3
I20260619 20:27:26.589576 25212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037: No bootstrap required, opened a new log
I20260619 20:27:26.591621 25212 raft_consensus.cc:348] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddd5ccf942e4f49b5c783962bb8f037" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34321 } }
I20260619 20:27:26.591818 25212 raft_consensus.cc:374] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:26.591856 25212 raft_consensus.cc:729] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bddd5ccf942e4f49b5c783962bb8f037, State: Initialized, Role: FOLLOWER
I20260619 20:27:26.592005 25212 consensus_queue.cc:260] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddd5ccf942e4f49b5c783962bb8f037" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34321 } }
I20260619 20:27:26.592088 25212 raft_consensus.cc:388] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:26.592118 25212 raft_consensus.cc:482] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:26.592160 25212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:26.592825 25212 raft_consensus.cc:504] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddd5ccf942e4f49b5c783962bb8f037" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34321 } }
I20260619 20:27:26.592926 25212 leader_election.cc:302] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bddd5ccf942e4f49b5c783962bb8f037; no voters:
I20260619 20:27:26.593060 25212 leader_election.cc:288] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:26.593109 25217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:26.593276 25212 sys_catalog.cc:564] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:26.593281 25217 raft_consensus.cc:686] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [term 1 LEADER]: Becoming Leader. State: Replica: bddd5ccf942e4f49b5c783962bb8f037, State: Running, Role: LEADER
I20260619 20:27:26.593488 25217 consensus_queue.cc:237] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddd5ccf942e4f49b5c783962bb8f037" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34321 } }
I20260619 20:27:26.593870 25219 sys_catalog.cc:455] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bddd5ccf942e4f49b5c783962bb8f037. Latest consensus state: current_term: 1 leader_uuid: "bddd5ccf942e4f49b5c783962bb8f037" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddd5ccf942e4f49b5c783962bb8f037" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34321 } } }
I20260619 20:27:26.594059 25219 sys_catalog.cc:458] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:26.594005 25218 sys_catalog.cc:455] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bddd5ccf942e4f49b5c783962bb8f037" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddd5ccf942e4f49b5c783962bb8f037" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34321 } } }
I20260619 20:27:26.594313 25218 sys_catalog.cc:458] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:26.594738 25226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:26.595402 25226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:26.596890 25226 catalog_manager.cc:1347] Generated new cluster ID: dda8a5f1e0ef49f29868a366b16a38ac
I20260619 20:27:26.596941 25226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:27:26 dist-test-slave-f9rk krb5kdc[25131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:26.601675 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:26.589861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.21.151.62:34321)
0619 20:27:26.590057 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:26.590067 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:26.590103 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:27:26.590856 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:26.590865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:26.590926 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:26.591083 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:26.591664 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:26.593316 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:26.594444 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:26.595072 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:26.595160 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:26.598163 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:26.598166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:26.598167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:26.598178 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:26.600118 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:26.600512 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:26.600515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:26.600516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:26.600624 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:26.600828 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:26.600828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:26.600829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:26.601105 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:26.601252 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:26.601363 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 20:27:26.601484 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260619 20:27:26.605419 25226 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:26.606036 25226 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:26.612357 25226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bddd5ccf942e4f49b5c783962bb8f037: Generated new TSK 0
I20260619 20:27:26.612524 25226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:27:26.642701 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:34321
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:26.743506 25238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:26.743736 25238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:26.743759 25238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:26.744906 25238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:26.744954 25238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:26.744982 25238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:26.746461 25238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:34321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:26.746685 25238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:26.746909 25238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:26.751915 25243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:26.751896 25244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:26.751909 25246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:26.752306 25238 server_base.cc:1034] running on GCE node
I20260619 20:27:26.752535 25238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:26.752836 25238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:26.753989 25238 hybrid_clock.cc:648] HybridClock initialized: now 1781900846753976 us; error 26 us; skew 500 ppm
Jun 19 20:27:26 dist-test-slave-f9rk krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900846, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:26.756440 25238 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:27:26.756872 25238 webserver.cc:460] Webserver started at http://127.21.151.1:38095/ using document root <none> and password file <none>
I20260619 20:27:26.757107 25238 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:26.757150 25238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:26.757308 25238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:26.758324 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "9a159f219dfc478cb8f85db97f2cbfc1"
format_stamp: "Formatted at 2026-06-19 20:27:26 on dist-test-slave-f9rk"
server_key: "c9cf6d75e86bbc383edcd9cdf76bc231"
server_key_iv: "7ce48ec02e68fb6bdcb060c172b6d6dc"
server_key_version: "encryptionkey@0"
I20260619 20:27:26.758688 25238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "9a159f219dfc478cb8f85db97f2cbfc1"
format_stamp: "Formatted at 2026-06-19 20:27:26 on dist-test-slave-f9rk"
server_key: "c9cf6d75e86bbc383edcd9cdf76bc231"
server_key_iv: "7ce48ec02e68fb6bdcb060c172b6d6dc"
server_key_version: "encryptionkey@0"
I20260619 20:27:26.760027 25238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:27:26.760808 25253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:26.760982 25238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:26.761075 25238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "9a159f219dfc478cb8f85db97f2cbfc1"
format_stamp: "Formatted at 2026-06-19 20:27:26 on dist-test-slave-f9rk"
server_key: "c9cf6d75e86bbc383edcd9cdf76bc231"
server_key_iv: "7ce48ec02e68fb6bdcb060c172b6d6dc"
server_key_version: "encryptionkey@0"
I20260619 20:27:26.761147 25238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:26.781860 25238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:26.783794 25238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:26.783950 25238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:26.784174 25238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:26.784483 25238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:26.784536 25238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:26.784578 25238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:26.784603 25238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:26.790112 25238 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:42777
I20260619 20:27:26.790185 25366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:42777 every 8 connection(s)
I20260619 20:27:26.790601 25238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:27:26 dist-test-slave-f9rk krb5kdc[25131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900846, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:26.797935 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 25238
I20260619 20:27:26.798028 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:26.798237 22108 external_mini_cluster.cc:1428] Setting key e3e5475fc241961214f6f3e7dd41e81b
I20260619 20:27:26.800609 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:26.791364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:34321 (local address 127.21.151.1:49263)
0619 20:27:26.791545 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:26.791585 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:26.792065 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:26.792260 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:26.792263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:26.792808 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:26.793014 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:26.793018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:26.793803 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:26.793804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:26.794696 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:26.794699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:26.795049 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:26.795050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:26.795174 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:26.795657 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:26.795665 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:26.796981 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:26.798457 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:26.798464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:26.798466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:26.798740 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:26.799679 (+ 939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:26.799680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:26.799680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:26.799778 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:26.800042 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:26.800050 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:26.800228 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 20:27:26.800319 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 20:27:26.800437 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260619 20:27:26.800607 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:26.791448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:49263 (local address 127.21.151.62:34321)
0619 20:27:26.791495 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:26.791498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:26.791679 (+ 181us) server_negotiation.cc:407] Connection header received
0619 20:27:26.792141 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:26.792142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:26.792151 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:26.792208 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:26.793103 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:26.793730 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:26.794771 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:26.794980 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:26.795042 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:26.797064 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:26.797065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:26.797066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:26.797073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:26.798386 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:26.798817 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:26.798818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:26.798819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:26.798879 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:26.799845 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:26.799846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:26.799847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:26.799956 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:26.800037 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:26.800376 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 20:27:26.800470 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:27:26.801083 25367 heartbeater.cc:347] Connected to a master server at 127.21.151.62:34321
I20260619 20:27:26.801194 25367 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:26.801380 25367 heartbeater.cc:511] Master 127.21.151.62:34321 requested a full tablet report, sending...
I20260619 20:27:26.801918 25176 ts_manager.cc:194] Registered new tserver with Master: 9a159f219dfc478cb8f85db97f2cbfc1 (127.21.151.1:42777)
I20260619 20:27:26.802904 25176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:49263
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:27:26.822351 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:34321
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:26.903052 25372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:26.903236 25372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:26.903257 25372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:26.904356 25372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:26.904388 25372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:26.904412 25372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:26.905730 25372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:34321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:26.905927 25372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:26.906085 25372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:26.910720 25377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:26.910748 25378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:26.910981 25372 server_base.cc:1034] running on GCE node
W20260619 20:27:26.910828 25380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:26.911234 25372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:26.911437 25372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:26.912566 25372 hybrid_clock.cc:648] HybridClock initialized: now 1781900846912550 us; error 41 us; skew 500 ppm
Jun 19 20:27:26 dist-test-slave-f9rk krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900846, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:26.914834 25372 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:27:26.915277 25372 webserver.cc:460] Webserver started at http://127.21.151.2:44447/ using document root <none> and password file <none>
I20260619 20:27:26.915473 25372 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:26.915514 25372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:26.915599 25372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:26.916633 25372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "1db2af609fd749c98a7459d6769f3055"
format_stamp: "Formatted at 2026-06-19 20:27:26 on dist-test-slave-f9rk"
server_key: "a943574bd4642396242dcf6314cb3cf5"
server_key_iv: "5146c3592fd09146c5849cbecc28a966"
server_key_version: "encryptionkey@0"
I20260619 20:27:26.916926 25372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "1db2af609fd749c98a7459d6769f3055"
format_stamp: "Formatted at 2026-06-19 20:27:26 on dist-test-slave-f9rk"
server_key: "a943574bd4642396242dcf6314cb3cf5"
server_key_iv: "5146c3592fd09146c5849cbecc28a966"
server_key_version: "encryptionkey@0"
I20260619 20:27:26.918095 25372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:26.918774 25387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:26.918959 25372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:26.919044 25372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "1db2af609fd749c98a7459d6769f3055"
format_stamp: "Formatted at 2026-06-19 20:27:26 on dist-test-slave-f9rk"
server_key: "a943574bd4642396242dcf6314cb3cf5"
server_key_iv: "5146c3592fd09146c5849cbecc28a966"
server_key_version: "encryptionkey@0"
I20260619 20:27:26.919109 25372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:26.939141 25372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:26.940989 25372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:26.941097 25372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:26.941273 25372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:26.941582 25372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:26.941614 25372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:26.941664 25372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:26.941696 25372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:26.947060 25372 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:42173
I20260619 20:27:26.947125 25500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:42173 every 8 connection(s)
I20260619 20:27:26.947429 25372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:27:26 dist-test-slave-f9rk krb5kdc[25131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900846, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:26.956152 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:26.948167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:59397 (local address 127.21.151.62:34321)
0619 20:27:26.948237 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:26.948240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:26.948430 (+ 190us) server_negotiation.cc:407] Connection header received
0619 20:27:26.948911 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:26.948912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:26.948922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:26.948980 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:26.949868 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:26.950368 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:26.951325 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:26.951518 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:26.951573 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:26.953634 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:26.953636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:26.953636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:26.953643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:26.954985 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:26.955334 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:26.955335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:26.955335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:26.955399 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:26.955608 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:26.955609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:26.955609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:26.955748 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:26.955800 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:26.956024 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 20:27:26.956085 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:27:26.956313 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:26.948140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:34321 (local address 127.21.151.2:59397)
0619 20:27:26.948302 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:26.948343 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:26.948827 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:26.949035 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:26.949057 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:26.949603 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:26.949793 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:26.949798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:26.950443 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:26.950444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:26.951247 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:26.951249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:26.951589 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:26.951589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:26.951756 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:26.952205 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:26.952213 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:26.953553 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:26.955052 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:26.955057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:26.955059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:26.955269 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:26.955459 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:26.955460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:26.955460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:26.955559 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:26.955800 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:26.955804 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:26.955916 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 20:27:26.956001 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:27:26.956117 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260619 20:27:26.956815 25501 heartbeater.cc:347] Connected to a master server at 127.21.151.62:34321
I20260619 20:27:26.956928 25501 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:26.956998 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 25372
I20260619 20:27:26.957114 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:26.957146 25501 heartbeater.cc:511] Master 127.21.151.62:34321 requested a full tablet report, sending...
I20260619 20:27:26.957297 22108 external_mini_cluster.cc:1428] Setting key 83697d61fe4e09bc0e07e5493ee116df
I20260619 20:27:26.957549 25176 ts_manager.cc:194] Registered new tserver with Master: 1db2af609fd749c98a7459d6769f3055 (127.21.151.2:42173)
I20260619 20:27:26.958098 25176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:59397
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:27:26.980124 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:34321
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:27.059247 25506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:27.059444 25506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:27.059468 25506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:27.060660 25506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:27.060703 25506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:27.060755 25506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:27.062305 25506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:34321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:27.062559 25506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:27.062778 25506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:27.068331 25511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:27.068439 25512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:27.068336 25514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:27.068792 25506 server_base.cc:1034] running on GCE node
I20260619 20:27:27.068989 25506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:27.069228 25506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:27.072484 25506 hybrid_clock.cc:648] HybridClock initialized: now 1781900847072441 us; error 35 us; skew 500 ppm
Jun 19 20:27:27 dist-test-slave-f9rk krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900847, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:27.075141 25506 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:27:27.075567 25506 webserver.cc:460] Webserver started at http://127.21.151.3:46323/ using document root <none> and password file <none>
I20260619 20:27:27.075822 25506 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:27.075870 25506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:27.076028 25506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:27.077132 25506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "9c02f575ec01417e8a31d335ee773748"
format_stamp: "Formatted at 2026-06-19 20:27:27 on dist-test-slave-f9rk"
server_key: "01cdf7d810d6a22714252c67d5fad3d9"
server_key_iv: "4507fd8fff0c017b0f791f7c48a1a7e8"
server_key_version: "encryptionkey@0"
I20260619 20:27:27.077483 25506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "9c02f575ec01417e8a31d335ee773748"
format_stamp: "Formatted at 2026-06-19 20:27:27 on dist-test-slave-f9rk"
server_key: "01cdf7d810d6a22714252c67d5fad3d9"
server_key_iv: "4507fd8fff0c017b0f791f7c48a1a7e8"
server_key_version: "encryptionkey@0"
I20260619 20:27:27.078738 25506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:27:27.079437 25521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:27.079602 25506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:27.079720 25506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "9c02f575ec01417e8a31d335ee773748"
format_stamp: "Formatted at 2026-06-19 20:27:27 on dist-test-slave-f9rk"
server_key: "01cdf7d810d6a22714252c67d5fad3d9"
server_key_iv: "4507fd8fff0c017b0f791f7c48a1a7e8"
server_key_version: "encryptionkey@0"
I20260619 20:27:27.079794 25506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:27.104245 25506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:27.106096 25506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:27.106249 25506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:27.106474 25506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:27.106770 25506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:27.106817 25506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:27.106859 25506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:27.106894 25506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:27.113021 25506 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:39755
I20260619 20:27:27.113063 25634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:39755 every 8 connection(s)
I20260619 20:27:27.113376 25506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:27.114466 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 25506
I20260619 20:27:27.114531 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:27.114686 22108 external_mini_cluster.cc:1428] Setting key 2be7ddf23afc880d3e0f064dffd0f9f3
Jun 19 20:27:27 dist-test-slave-f9rk krb5kdc[25131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900847, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:27.122246 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:27.114040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:58613 (local address 127.21.151.62:34321)
0619 20:27:27.114128 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:27.114131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:27.114295 (+ 164us) server_negotiation.cc:407] Connection header received
0619 20:27:27.114874 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:27.114876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:27.114888 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:27.114956 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:27.116099 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:27.116590 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:27.117529 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:27.117715 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:27.117774 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:27.119872 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:27.119873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:27.119874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:27.119880 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:27.121079 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:27.121418 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:27.121419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:27.121420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:27.121469 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:27.121715 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:27.121716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:27.121716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:27.121826 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:27.121897 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:27.122124 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 20:27:27.122182 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 20:27:27.122407 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:27.114008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:34321 (local address 127.21.151.3:58613)
0619 20:27:27.114182 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:27.114217 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:27.114783 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:27.115036 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:27.115041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:27.115794 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:27.116005 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:27.116010 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:27.116666 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:27.116666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:27.117454 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:27.117456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:27.117785 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:27.117785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:27.117881 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:27.118427 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:27.118446 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:27.119801 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:27.121141 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:27.121147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:27.121148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:27.121352 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:27.121520 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:27.121521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:27.121521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:27.121651 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:27.121895 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:27.121899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:27.122019 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 20:27:27.122102 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 20:27:27.122215 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260619 20:27:27.122926 25635 heartbeater.cc:347] Connected to a master server at 127.21.151.62:34321
I20260619 20:27:27.123004 25635 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:27.123142 25635 heartbeater.cc:511] Master 127.21.151.62:34321 requested a full tablet report, sending...
I20260619 20:27:27.123502 25176 ts_manager.cc:194] Registered new tserver with Master: 9c02f575ec01417e8a31d335ee773748 (127.21.151.3:39755)
I20260619 20:27:27.123912 25176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:58613
I20260619 20:27:27.126397 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:27:27 dist-test-slave-f9rk krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900847, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 20:27:27 dist-test-slave-f9rk krb5kdc[25131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900847, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:27.147657 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:27.141154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.21.151.62:34321)
0619 20:27:27.141207 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:27.141209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:27.141316 (+ 107us) server_negotiation.cc:407] Connection header received
0619 20:27:27.141376 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:27.141377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:27.141385 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:27.141440 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:27.141955 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:27.142439 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:27.143124 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:27.143295 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:27.143354 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:27.145562 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:27.145563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:27.145564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:27.145569 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:27.146707 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:27.147000 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:27.147001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:27.147001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:27.147047 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:27.147196 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:27.147197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:27.147197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:27.147329 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:27.147403 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:27.147480 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 20:27:27.147546 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 20:27:27.147873 25176 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56874
I20260619 20:27:27.148506 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 25238
I20260619 20:27:27.153071 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 25372
I20260619 20:27:27.157755 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 25506
I20260619 20:27:27.161764 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 25144
2026-06-19T20:27:27Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1825 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:27 dist-test-slave-f9rk krb5kdc[25647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:27 dist-test-slave-f9rk krb5kdc[25647](info): set up 2 sockets
Jun 19 20:27:27 dist-test-slave-f9rk krb5kdc[25647](info): commencing operation
krb5kdc: starting...
W20260619 20:27:28.200507 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:28.310514 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:41075
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:41075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:28.388998 25660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:28.389231 25660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:28.389268 25660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:28.390420 25660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:28.390496 25660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:28.390524 25660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:28.390571 25660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:28.390594 25660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:28.392097 25660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:41075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:41075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:28.392365 25660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:28.392608 25660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:28.397230 25668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:28.397230 25666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:28.397444 25665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:28.397704 25660 server_base.cc:1034] running on GCE node
I20260619 20:27:28.397981 25660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:28.398296 25660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:28.399472 25660 hybrid_clock.cc:648] HybridClock initialized: now 1781900848399457 us; error 32 us; skew 500 ppm
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:28.404979 25660 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:28.405409 25660 webserver.cc:460] Webserver started at http://127.21.151.62:46451/ using document root <none> and password file <none>
I20260619 20:27:28.405624 25660 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:28.405685 25660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:28.405825 25660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:28.406862 25660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "dc17bc606f1f4529991a92529beb2138"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "0132006de4424f87b3f43ab45885652e"
server_key_iv: "a09c5a08519f9b4ddb41df3b5a97fba4"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.407208 25660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "dc17bc606f1f4529991a92529beb2138"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "0132006de4424f87b3f43ab45885652e"
server_key_iv: "a09c5a08519f9b4ddb41df3b5a97fba4"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.408538 25660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:28.409179 25675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:28.409327 25660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:28.409443 25660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "dc17bc606f1f4529991a92529beb2138"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "0132006de4424f87b3f43ab45885652e"
server_key_iv: "a09c5a08519f9b4ddb41df3b5a97fba4"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.409518 25660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:28.425982 25660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:28.428030 25660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:28.428187 25660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:28.431973 25660 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:41075
I20260619 20:27:28.432034 25727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:41075 every 8 connection(s)
I20260619 20:27:28.432592 25660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:28.433270 25728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:28.434931 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 25660
I20260619 20:27:28.435019 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:28.435196 22108 external_mini_cluster.cc:1428] Setting key 2b182a47ce6865ad99de109e72af4f04
I20260619 20:27:28.435777 25728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138: Bootstrap starting.
I20260619 20:27:28.436400 25728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:28.436780 25728 log.cc:824] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:28.437608 25728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138: No bootstrap required, opened a new log
I20260619 20:27:28.439795 25728 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc17bc606f1f4529991a92529beb2138" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41075 } }
I20260619 20:27:28.439929 25728 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:28.439952 25728 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc17bc606f1f4529991a92529beb2138, State: Initialized, Role: FOLLOWER
I20260619 20:27:28.440099 25728 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc17bc606f1f4529991a92529beb2138" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41075 } }
I20260619 20:27:28.440171 25728 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:28.440203 25728 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:28.440250 25728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:28.440940 25728 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc17bc606f1f4529991a92529beb2138" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41075 } }
I20260619 20:27:28.441056 25728 leader_election.cc:302] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc17bc606f1f4529991a92529beb2138; no voters:
I20260619 20:27:28.441226 25728 leader_election.cc:288] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:28.441324 25733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:28.441481 25733 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [term 1 LEADER]: Becoming Leader. State: Replica: dc17bc606f1f4529991a92529beb2138, State: Running, Role: LEADER
I20260619 20:27:28.441506 25728 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:28.441637 25733 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc17bc606f1f4529991a92529beb2138" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41075 } }
I20260619 20:27:28.441973 25735 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc17bc606f1f4529991a92529beb2138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc17bc606f1f4529991a92529beb2138" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41075 } } }
I20260619 20:27:28.442132 25734 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc17bc606f1f4529991a92529beb2138. Latest consensus state: current_term: 1 leader_uuid: "dc17bc606f1f4529991a92529beb2138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc17bc606f1f4529991a92529beb2138" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 41075 } } }
I20260619 20:27:28.442198 25734 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:28.442175 25735 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:28.442436 25742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:28.442935 25742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:28.444677 25742 catalog_manager.cc:1347] Generated new cluster ID: 73892c53551b437c8230bb0979574f9c
I20260619 20:27:28.444747 25742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:28.447075 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:28.435732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.21.151.62:41075)
0619 20:27:28.435933 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:28.435942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:28.435968 (+ 26us) server_negotiation.cc:407] Connection header received
0619 20:27:28.436394 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:28.436400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:28.436443 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:28.436617 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:28.437576 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.438473 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.439607 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.440139 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.440258 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:28.443072 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:28.443076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:28.443076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:28.443086 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:28.445140 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.445515 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.445518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.445518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.445615 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.445804 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.445804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.445805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.446099 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:28.446232 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:28.446726 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 20:27:28.446867 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260619 20:27:28.463644 25742 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:28.464190 25742 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:28.473006 25742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc17bc606f1f4529991a92529beb2138: Generated new TSK 0
I20260619 20:27:28.473206 25742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:27:28.497681 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41075
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:28.576838 25754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:28.577064 25754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:28.577087 25754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:28.578300 25754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:28.578336 25754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:28.578359 25754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:28.579798 25754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:28.579989 25754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:28.580168 25754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:28.585120 25762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:28.585108 25759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:28.585105 25760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:28.585225 25754 server_base.cc:1034] running on GCE node
I20260619 20:27:28.585582 25754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:28.585815 25754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:28.586953 25754 hybrid_clock.cc:648] HybridClock initialized: now 1781900848586932 us; error 41 us; skew 500 ppm
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:28.589350 25754 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:27:28.589747 25754 webserver.cc:460] Webserver started at http://127.21.151.1:45489/ using document root <none> and password file <none>
I20260619 20:27:28.589951 25754 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:28.590013 25754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:28.590157 25754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:28.591166 25754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "8d6b0b038ca6484587be5fb8db4766b9"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "80b50089eb1cca83a68b2b59c89b1e89"
server_key_iv: "833e42fc1eeb16a64baba252e4e8c66a"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.591516 25754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "8d6b0b038ca6484587be5fb8db4766b9"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "80b50089eb1cca83a68b2b59c89b1e89"
server_key_iv: "833e42fc1eeb16a64baba252e4e8c66a"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.592804 25754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:27:28.593469 25769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:28.593626 25754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:28.593734 25754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "8d6b0b038ca6484587be5fb8db4766b9"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "80b50089eb1cca83a68b2b59c89b1e89"
server_key_iv: "833e42fc1eeb16a64baba252e4e8c66a"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.593801 25754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:28.626401 25754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:28.628280 25754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:28.628441 25754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:28.628669 25754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:28.628971 25754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:28.629024 25754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:28.629060 25754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:28.629084 25754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:28.634486 25754 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:43943
I20260619 20:27:28.634553 25882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:43943 every 8 connection(s)
I20260619 20:27:28.634860 25754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:28.642292 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 25754
I20260619 20:27:28.642401 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:28.642580 22108 external_mini_cluster.cc:1428] Setting key aa9f2aa3c136e0a98ca10173e2b134a3
I20260619 20:27:28.643870 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:28.635532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:40885 (local address 127.21.151.62:41075)
0619 20:27:28.635579 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:28.635581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:28.635782 (+ 201us) server_negotiation.cc:407] Connection header received
0619 20:27:28.636656 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:28.636657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:28.636665 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:28.636726 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:28.637313 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.637812 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.638760 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.638946 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.638995 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:28.640995 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:28.640996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:28.640996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:28.641002 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:28.642315 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.642828 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.642829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.642829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.642895 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.643182 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.643183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.643183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.643311 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:28.643383 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:28.643644 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 20:27:28.643732 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:27:28.643895 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:28.635485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:41075 (local address 127.21.151.1:40885)
0619 20:27:28.635659 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:28.635698 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:28.636572 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:28.636787 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:28.636791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:28.637029 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:28.637229 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:28.637233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.637884 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.637885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:28.638687 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:28.638689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.639012 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.639013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:28.639121 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:28.639592 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:28.639600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:28.640909 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:28.642426 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:28.642433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:28.642434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:28.642746 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:28.642967 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:28.642968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:28.642968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:28.643112 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:28.643376 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:28.643381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:28.643488 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 20:27:28.643574 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 20:27:28.643747 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260619 20:27:28.644879 25883 heartbeater.cc:347] Connected to a master server at 127.21.151.62:41075
I20260619 20:27:28.644960 25883 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:28.645143 25883 heartbeater.cc:511] Master 127.21.151.62:41075 requested a full tablet report, sending...
I20260619 20:27:28.645704 25692 ts_manager.cc:194] Registered new tserver with Master: 8d6b0b038ca6484587be5fb8db4766b9 (127.21.151.1:43943)
I20260619 20:27:28.646311 25692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:40885
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:27:28.666097 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41075
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:28.744566 25888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:28.744757 25888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:28.744781 25888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:28.745839 25888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:28.745893 25888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:28.745919 25888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:28.747278 25888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:28.747475 25888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:28.747686 25888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:27:28.752543 25888 server_base.cc:1034] running on GCE node
W20260619 20:27:28.752583 25893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:28.752617 25894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:28.752751 25896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:28.752936 25888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:28.753180 25888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:28.754330 25888 hybrid_clock.cc:648] HybridClock initialized: now 1781900848754315 us; error 27 us; skew 500 ppm
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:28.756639 25888 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:27:28.757058 25888 webserver.cc:460] Webserver started at http://127.21.151.2:45365/ using document root <none> and password file <none>
I20260619 20:27:28.757268 25888 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:28.757325 25888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:28.757480 25888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:28.758415 25888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "cfed8ba806cf4897bfffde667875a5ba"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "6f05b207f592896aaea26e922d2ea404"
server_key_iv: "fa88b15e1dc2f89573df2c33b6807b3d"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.758751 25888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "cfed8ba806cf4897bfffde667875a5ba"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "6f05b207f592896aaea26e922d2ea404"
server_key_iv: "fa88b15e1dc2f89573df2c33b6807b3d"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.760087 25888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:27:28.760775 25903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:28.760933 25888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:28.761034 25888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "cfed8ba806cf4897bfffde667875a5ba"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "6f05b207f592896aaea26e922d2ea404"
server_key_iv: "fa88b15e1dc2f89573df2c33b6807b3d"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.761102 25888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:28.768435 25888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:28.770117 25888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:28.770277 25888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:28.770524 25888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:28.770859 25888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:28.770910 25888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:28.770955 25888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:28.770999 25888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:28.776724 25888 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:32861
I20260619 20:27:28.776767 26016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:32861 every 8 connection(s)
I20260619 20:27:28.777064 25888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:27:28.779771 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 25888
I20260619 20:27:28.779873 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:28.780030 22108 external_mini_cluster.cc:1428] Setting key 452f982ddfb8a340848844b807048e2e
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:28.787166 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:28.777678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:42085 (local address 127.21.151.62:41075)
0619 20:27:28.777730 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:28.777732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:28.777904 (+ 172us) server_negotiation.cc:407] Connection header received
0619 20:27:28.778807 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:28.778808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:28.778821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:28.778869 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:28.779451 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.780208 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.781623 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.781913 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.781984 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:28.784470 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:28.784471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:28.784471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:28.784477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:28.785811 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.786288 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.786289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.786289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.786335 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.786570 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.786571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.786571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.786663 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:28.786709 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:28.787045 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 20:27:28.787103 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 20:27:28.787456 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:28.777666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:41075 (local address 127.21.151.2:42085)
0619 20:27:28.777802 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:28.777834 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:28.778697 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:28.778937 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:28.778940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:28.779181 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:28.779352 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:28.779356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.780307 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.780308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:28.781536 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:28.781539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.782051 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.782052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:28.782166 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:28.782662 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:28.782675 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:28.784387 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:28.785890 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:28.785897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:28.785899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:28.786210 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:28.786377 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:28.786378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:28.786379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:28.786512 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:28.786722 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:28.786728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:28.786892 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 20:27:28.787033 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 20:27:28.787201 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260619 20:27:28.787959 26017 heartbeater.cc:347] Connected to a master server at 127.21.151.62:41075
I20260619 20:27:28.788061 26017 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:28.788254 26017 heartbeater.cc:511] Master 127.21.151.62:41075 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
I20260619 20:27:28.788617 25692 ts_manager.cc:194] Registered new tserver with Master: cfed8ba806cf4897bfffde667875a5ba (127.21.151.2:32861)
I20260619 20:27:28.788973 25692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:42085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:27:28.804572 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41075
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:28.883417 26022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:28.883675 26022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:28.883711 26022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:28.884800 26022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:28.884835 26022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:28.884859 26022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:28.886649 26022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:41075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:28.886837 26022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:28.887022 26022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:28.891752 26028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:28.891710 26027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:28.891773 26030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:28.892033 26022 server_base.cc:1034] running on GCE node
I20260619 20:27:28.892256 26022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:28.892474 26022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:28.893647 26022 hybrid_clock.cc:648] HybridClock initialized: now 1781900848893649 us; error 41 us; skew 500 ppm
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:28.896167 26022 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:27:28.896638 26022 webserver.cc:460] Webserver started at http://127.21.151.3:35909/ using document root <none> and password file <none>
I20260619 20:27:28.896875 26022 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:28.896934 26022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:28.897068 26022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:28.898089 26022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "96c5bbefa6bf40dd8d17a75a739a4e54"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "cb41a7ebf4aed07e153752e6c8343dd8"
server_key_iv: "e6c79ca132cd13f51e8ca789fcf1a5f0"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.898414 26022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "96c5bbefa6bf40dd8d17a75a739a4e54"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "cb41a7ebf4aed07e153752e6c8343dd8"
server_key_iv: "e6c79ca132cd13f51e8ca789fcf1a5f0"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.899689 26022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:27:28.900400 26037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:28.900560 26022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:28.900661 26022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "96c5bbefa6bf40dd8d17a75a739a4e54"
format_stamp: "Formatted at 2026-06-19 20:27:28 on dist-test-slave-f9rk"
server_key: "cb41a7ebf4aed07e153752e6c8343dd8"
server_key_iv: "e6c79ca132cd13f51e8ca789fcf1a5f0"
server_key_version: "encryptionkey@0"
I20260619 20:27:28.900741 26022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:28.917544 26022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:28.919806 26022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:28.919960 26022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:28.920188 26022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:28.920481 26022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:28.920532 26022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:28.920571 26022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:28.920610 26022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:28.926115 26022 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:45783
I20260619 20:27:28.926185 26150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:45783 every 8 connection(s)
I20260619 20:27:28.926492 26022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:28.928925 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 26022
I20260619 20:27:28.929042 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:28.929205 22108 external_mini_cluster.cc:1428] Setting key e16b8dc1de84fa543f1d78cce21e17f2
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:28.935715 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:28.927277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:52631 (local address 127.21.151.62:41075)
0619 20:27:28.927355 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:28.927357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:28.927530 (+ 173us) server_negotiation.cc:407] Connection header received
0619 20:27:28.928552 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:28.928553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:28.928565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:28.928618 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:28.929269 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.930031 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.930987 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.931194 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.931265 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:28.933334 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:28.933335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:28.933335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:28.933342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:28.934659 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.934981 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.934981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.934982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.935035 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.935219 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.935219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.935219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.935309 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:28.935355 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:28.935563 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 20:27:28.935650 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 20:27:28.935902 26153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:28.927181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:41075 (local address 127.21.151.3:52631)
0619 20:27:28.927386 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:28.927431 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:28.928393 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:28.928683 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:28.928687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:28.928932 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:28.929184 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:28.929189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.930101 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.930102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:28.930892 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:28.930894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.931269 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.931270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:28.931394 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:28.931932 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:28.931940 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:28.933246 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:28.934721 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:28.934725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:28.934727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:28.934919 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:28.935084 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:28.935085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:28.935085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:28.935171 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:28.935356 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:28.935359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:28.935462 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 20:27:28.935539 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:27:28.935696 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":107,"threads_started":1}
I20260619 20:27:28.936321 26151 heartbeater.cc:347] Connected to a master server at 127.21.151.62:41075
I20260619 20:27:28.936408 26151 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:28.936542 26151 heartbeater.cc:511] Master 127.21.151.62:41075 requested a full tablet report, sending...
I20260619 20:27:28.936861 25692 ts_manager.cc:194] Registered new tserver with Master: 96c5bbefa6bf40dd8d17a75a739a4e54 (127.21.151.3:45783)
I20260619 20:27:28.937204 25692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:52631
I20260619 20:27:28.940696 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.1@KRBTEST.COM
I20260619 20:27:28.962518 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:28.955263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.21.151.1:43943)
0619 20:27:28.955434 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:28.955439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:28.955449 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:28.955475 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:28.955476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:28.955490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:28.955554 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:28.956112 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.956790 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.957662 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.957956 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.958018 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:28.960100 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:28.960107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:28.960107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:28.960113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:28.961627 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.961911 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.961914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.961914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.961966 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.962098 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.962098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.962099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.962203 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:28.962295 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:28.962354 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 20:27:28.962407 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
Jun 19 20:27:28 dist-test-slave-f9rk krb5kdc[25647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:28.969954 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:28.963768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.21.151.62:41075)
0619 20:27:28.963808 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:28.963809 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:28.963834 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:27:28.963895 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:28.963896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:28.963904 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:28.963946 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:28.964397 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.964940 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.965672 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:28.965847 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:28.965898 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:28.967812 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:28.967814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:28.967814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:28.967820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:28.969108 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.969378 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.969379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.969379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.969425 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:28.969572 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:28.969572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:28.969573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:28.969692 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:28.969757 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:28.969840 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 20:27:28.969893 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 20:27:28.970170 25692 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36012
I20260619 20:27:28.970482 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 25754
I20260619 20:27:28.975407 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 25888
I20260619 20:27:28.980434 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 26022
I20260619 20:27:28.984519 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 25660
2026-06-19T20:27:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1822 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:29 dist-test-slave-f9rk krb5kdc[26163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:29 dist-test-slave-f9rk krb5kdc[26163](info): set up 2 sockets
Jun 19 20:27:29 dist-test-slave-f9rk krb5kdc[26163](info): commencing operation
krb5kdc: starting...
W20260619 20:27:30.261057 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:30 dist-test-slave-f9rk krb5kdc[26163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:30.373816 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:39351
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:39351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:30.453639 26176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:30.453888 26176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:30.453924 26176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:30.455031 26176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:30.455094 26176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:30.455117 26176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:30.455137 26176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:30.455164 26176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:30.456595 26176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:39351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:39351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:30.456895 26176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:30.457114 26176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:30.461920 26182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:30.461995 26184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:30.462157 26181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:30.462234 26176 server_base.cc:1034] running on GCE node
I20260619 20:27:30.462409 26176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:30.462687 26176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:30.463819 26176 hybrid_clock.cc:648] HybridClock initialized: now 1781900850463805 us; error 28 us; skew 500 ppm
Jun 19 20:27:30 dist-test-slave-f9rk krb5kdc[26163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900850, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:30.466226 26176 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:30.466495 26176 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:30.466544 26176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:30.466634 26176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:30.467608 26176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "f6c1886d01f640e4a5a4022cada36dd4"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "d5c8a489d1735cb84e4545b0239bea53"
server_key_iv: "06e4758b017f1fd81a142d929ed7fca0"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.467975 26176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "f6c1886d01f640e4a5a4022cada36dd4"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "d5c8a489d1735cb84e4545b0239bea53"
server_key_iv: "06e4758b017f1fd81a142d929ed7fca0"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.469255 26176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:30.469895 26190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:30.470053 26176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:30.470156 26176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "f6c1886d01f640e4a5a4022cada36dd4"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "d5c8a489d1735cb84e4545b0239bea53"
server_key_iv: "06e4758b017f1fd81a142d929ed7fca0"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.470224 26176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:30.489199 26176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:30.491488 26176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:30.491685 26176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:30.495535 26176 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:39351
I20260619 20:27:30.495569 26242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:39351 every 8 connection(s)
I20260619 20:27:30.495880 26176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:30.496806 26243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:30.498257 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 26176
I20260619 20:27:30.498364 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:30.498538 22108 external_mini_cluster.cc:1428] Setting key ffe28ea3fb597692646f6f9a09b1c079
I20260619 20:27:30.499433 26243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4: Bootstrap starting.
I20260619 20:27:30.500149 26243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:30.500453 26243 log.cc:824] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:30.501190 26243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4: No bootstrap required, opened a new log
I20260619 20:27:30.503285 26243 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c1886d01f640e4a5a4022cada36dd4" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39351 } }
I20260619 20:27:30.503449 26243 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:30.503539 26243 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c1886d01f640e4a5a4022cada36dd4, State: Initialized, Role: FOLLOWER
I20260619 20:27:30.503705 26243 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c1886d01f640e4a5a4022cada36dd4" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39351 } }
I20260619 20:27:30.503798 26243 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:30.503829 26243 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:30.503875 26243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:30.504401 26243 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c1886d01f640e4a5a4022cada36dd4" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39351 } }
I20260619 20:27:30.504482 26243 leader_election.cc:302] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6c1886d01f640e4a5a4022cada36dd4; no voters:
I20260619 20:27:30.504673 26243 leader_election.cc:288] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:30.504717 26248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:30.504864 26248 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [term 1 LEADER]: Becoming Leader. State: Replica: f6c1886d01f640e4a5a4022cada36dd4, State: Running, Role: LEADER
I20260619 20:27:30.504915 26243 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:30.505040 26248 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c1886d01f640e4a5a4022cada36dd4" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39351 } }
Jun 19 20:27:30 dist-test-slave-f9rk krb5kdc[26163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:30.506040 26249 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6c1886d01f640e4a5a4022cada36dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c1886d01f640e4a5a4022cada36dd4" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39351 } } }
I20260619 20:27:30.506137 26249 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:30.506433 26250 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6c1886d01f640e4a5a4022cada36dd4. Latest consensus state: current_term: 1 leader_uuid: "f6c1886d01f640e4a5a4022cada36dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c1886d01f640e4a5a4022cada36dd4" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39351 } } }
I20260619 20:27:30.506506 26250 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4 [sys.catalog]: This master's current role is: LEADER
W20260619 20:27:30.506769 26264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:27:30.506875 26264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:27:30.506951 26262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:30.507143 26262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:30.508612 26262 catalog_manager.cc:1347] Generated new cluster ID: 0d75074b3629457a991c0724c2a0bb20
I20260619 20:27:30.508670 26262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:30.509989 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:30.499046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.21.151.62:39351)
0619 20:27:30.499245 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:30.499255 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:30.499287 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:27:30.499843 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:30.499852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:30.499915 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:30.500067 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:30.500690 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:30.501995 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:30.502723 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:30.503387 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:30.503496 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:30.506188 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:30.506192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:30.506193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:30.506205 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:30.508140 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:30.508545 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:30.508548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:30.508549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:30.508643 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:30.508833 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:30.508834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:30.508835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:30.509122 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:30.509256 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:30.509672 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 20:27:30.509801 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260619 20:27:30.521246 26262 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:30.521910 26262 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:30.526283 26262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4: Generated new TSK 0
I20260619 20:27:30.526453 26262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:27:30.558777 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39351
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:30.638033 26269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:30.638216 26269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:30.638237 26269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:30.639312 26269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:30.639345 26269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:30.639370 26269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:30.640790 26269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:30.640969 26269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:30.641139 26269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:30.645776 26274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:30.645772 26275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:30.645772 26277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:30.646170 26269 server_base.cc:1034] running on GCE node
I20260619 20:27:30.646404 26269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:30.646629 26269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:30.647797 26269 hybrid_clock.cc:648] HybridClock initialized: now 1781900850647787 us; error 35 us; skew 500 ppm
Jun 19 20:27:30 dist-test-slave-f9rk krb5kdc[26163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900850, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:30.650246 26269 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:27:30.650609 26269 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:30.650677 26269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:30.650822 26269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:30.651924 26269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "fbac29b8f0e840b895d4547e80f1285d"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "72fe0770a3ae9a953880f65ae7274cc7"
server_key_iv: "f06623400cc785cde270c6ca1c519144"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.652271 26269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "fbac29b8f0e840b895d4547e80f1285d"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "72fe0770a3ae9a953880f65ae7274cc7"
server_key_iv: "f06623400cc785cde270c6ca1c519144"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.653631 26269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:30.654376 26283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:30.654563 26269 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:30.654660 26269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "fbac29b8f0e840b895d4547e80f1285d"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "72fe0770a3ae9a953880f65ae7274cc7"
server_key_iv: "f06623400cc785cde270c6ca1c519144"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.654731 26269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:30.675110 26269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:30.677280 26269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:30.677426 26269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:30.677644 26269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:30.677934 26269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:30.677978 26269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:30.678031 26269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:30.678061 26269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:30.683959 26269 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:36873
I20260619 20:27:30.684053 26396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:36873 every 8 connection(s)
I20260619 20:27:30.684283 26269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:27:30 dist-test-slave-f9rk krb5kdc[26163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900850, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:30.693157 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:30.684931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:41271 (local address 127.21.151.62:39351)
0619 20:27:30.684997 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:30.684999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:30.685236 (+ 237us) server_negotiation.cc:407] Connection header received
0619 20:27:30.685687 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:30.685688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:30.685698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:30.685745 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:30.686659 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:30.687132 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:30.688182 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:30.688386 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:30.688435 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:30.690575 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:30.690576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:30.690577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:30.690584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:30.691949 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:30.692267 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:30.692267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:30.692268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:30.692313 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:30.692642 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:30.692643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:30.692643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:30.692736 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:30.692785 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:30.693035 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 20:27:30.693096 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:27:30.693369 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:30.684931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39351 (local address 127.21.151.1:41271)
0619 20:27:30.685120 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:30.685154 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:30.685607 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:30.685810 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:30.685813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:30.686064 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:30.686569 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:30.686574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:30.687204 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:30.687205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:30.688109 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:30.688111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:30.688451 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:30.688452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:30.688545 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:30.688990 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:30.688998 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:30.690498 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:30.692016 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:30.692021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:30.692022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:30.692211 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:30.692472 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:30.692473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:30.692473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:30.692580 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:30.692792 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:30.692796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:30.692901 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 20:27:30.693013 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 20:27:30.693126 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":106,"threads_started":1}
I20260619 20:27:30.693581 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 26269
I20260619 20:27:30.693701 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:30.693895 22108 external_mini_cluster.cc:1428] Setting key 58d42d5a8984b0bf12aadc70cd0d66ed
I20260619 20:27:30.693914 26397 heartbeater.cc:347] Connected to a master server at 127.21.151.62:39351
I20260619 20:27:30.693998 26397 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:30.694175 26397 heartbeater.cc:511] Master 127.21.151.62:39351 requested a full tablet report, sending...
I20260619 20:27:30.694707 26207 ts_manager.cc:194] Registered new tserver with Master: fbac29b8f0e840b895d4547e80f1285d (127.21.151.1:36873)
I20260619 20:27:30.695341 26207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:41271
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:27:30.717742 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39351
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:30.795709 26402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:30.795892 26402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:30.795914 26402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:30.796940 26402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:30.796972 26402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:30.796994 26402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:30.798409 26402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:30.798607 26402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:30.798774 26402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:27:30.803488 26402 server_base.cc:1034] running on GCE node
W20260619 20:27:30.803653 26407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:30.803638 26410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:30.803835 26408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:30.804080 26402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:30.804312 26402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:30.805461 26402 hybrid_clock.cc:648] HybridClock initialized: now 1781900850805442 us; error 25 us; skew 500 ppm
Jun 19 20:27:30 dist-test-slave-f9rk krb5kdc[26163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900850, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:30.807890 26402 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:27:30.808213 26402 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:30.808279 26402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:30.808414 26402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:30.809350 26402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "d35315045328494eb30b9ed0feb20980"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "f0bd90156ee806f8fead9f4ef696f885"
server_key_iv: "144fc8d2211e8a51b133776fa8b9207e"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.809679 26402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "d35315045328494eb30b9ed0feb20980"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "f0bd90156ee806f8fead9f4ef696f885"
server_key_iv: "144fc8d2211e8a51b133776fa8b9207e"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.810874 26402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:30.811539 26416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:30.811705 26402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:30.811813 26402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "d35315045328494eb30b9ed0feb20980"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "f0bd90156ee806f8fead9f4ef696f885"
server_key_iv: "144fc8d2211e8a51b133776fa8b9207e"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.811879 26402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:30.845374 26402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:30.847107 26402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:30.847259 26402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:30.847494 26402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:30.847820 26402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:30.847877 26402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:30.847909 26402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:30.847935 26402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:30.853467 26402 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:38183
I20260619 20:27:30.853497 26529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:38183 every 8 connection(s)
I20260619 20:27:30.853787 26402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:27:30 dist-test-slave-f9rk krb5kdc[26163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900850, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:30.862206 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 26402
I20260619 20:27:30.862308 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:30.862397 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:30.854441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:35141 (local address 127.21.151.62:39351)
0619 20:27:30.854512 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:30.854514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:30.854694 (+ 180us) server_negotiation.cc:407] Connection header received
0619 20:27:30.855202 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:30.855203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:30.855211 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:30.855265 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:30.856212 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:30.856679 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:30.857676 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:30.857854 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:30.857907 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:30.859885 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:30.859886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:30.859886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:30.859893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:30.861218 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:30.861555 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:30.861556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:30.861556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:30.861602 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:30.861810 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:30.861810 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:30.861810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:30.861901 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:30.861947 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:30.862195 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 20:27:30.862278 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:27:30.862540 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:30.854413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39351 (local address 127.21.151.2:35141)
0619 20:27:30.854569 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:30.854609 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:30.855117 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:30.855325 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:30.855328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:30.855570 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:30.856137 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:30.856142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:30.856748 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:30.856748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:30.857604 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:30.857607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:30.857920 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:30.857921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:30.858026 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:30.858459 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:30.858468 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:30.859798 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:30.861282 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:30.861288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:30.861289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:30.861495 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:30.861654 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:30.861655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:30.861655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:30.861747 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:30.861948 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:30.861952 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:30.862059 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 20:27:30.862139 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 20:27:30.862249 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260619 20:27:30.862552 22108 external_mini_cluster.cc:1428] Setting key da97ba3f44c22cd2d487b564dcbcd2af
I20260619 20:27:30.863157 26530 heartbeater.cc:347] Connected to a master server at 127.21.151.62:39351
I20260619 20:27:30.863261 26530 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:30.863420 26530 heartbeater.cc:511] Master 127.21.151.62:39351 requested a full tablet report, sending...
I20260619 20:27:30.863823 26207 ts_manager.cc:194] Registered new tserver with Master: d35315045328494eb30b9ed0feb20980 (127.21.151.2:38183)
I20260619 20:27:30.864205 26207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:35141
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:27:30.886751 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39351
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:30.964504 26535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:30.964747 26535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:30.964771 26535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:30.965922 26535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:30.965967 26535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:30.966005 26535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:30.967460 26535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:30.967686 26535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:30.967885 26535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:30.972759 26543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:30.972766 26540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:30.972767 26541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:30.973033 26535 server_base.cc:1034] running on GCE node
I20260619 20:27:30.973248 26535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:30.973467 26535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:30.974632 26535 hybrid_clock.cc:648] HybridClock initialized: now 1781900850974617 us; error 30 us; skew 500 ppm
Jun 19 20:27:30 dist-test-slave-f9rk krb5kdc[26163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900850, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:30.977007 26535 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:27:30.977380 26535 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:30.977473 26535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:30.977612 26535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:30.978542 26535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "85b18525f34c4de68d94ae7169522968"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "314ff60e3fe7e501dbc0d2a9ba9fa36f"
server_key_iv: "1af2004c547777e23fd08c7ef12c918f"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.978868 26535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "85b18525f34c4de68d94ae7169522968"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "314ff60e3fe7e501dbc0d2a9ba9fa36f"
server_key_iv: "1af2004c547777e23fd08c7ef12c918f"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.980202 26535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 20:27:30.980892 26549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:30.981082 26535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:30.981180 26535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "85b18525f34c4de68d94ae7169522968"
format_stamp: "Formatted at 2026-06-19 20:27:30 on dist-test-slave-f9rk"
server_key: "314ff60e3fe7e501dbc0d2a9ba9fa36f"
server_key_iv: "1af2004c547777e23fd08c7ef12c918f"
server_key_version: "encryptionkey@0"
I20260619 20:27:30.981249 26535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:31.001124 26535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:31.002923 26535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:31.003093 26535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:31.003340 26535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:31.003690 26535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:31.003736 26535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.003778 26535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:31.003806 26535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.009819 26535 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:33709
I20260619 20:27:31.009846 26662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:33709 every 8 connection(s)
I20260619 20:27:31.010174 26535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:31.011298 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 26535
I20260619 20:27:31.011366 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:31.011528 22108 external_mini_cluster.cc:1428] Setting key 1b65dc2415cdcf2bf1eaf88390b58945
Jun 19 20:27:31 dist-test-slave-f9rk krb5kdc[26163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900850, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:31.019090 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.010838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:56391 (local address 127.21.151.62:39351)
0619 20:27:31.010878 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.010880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.011105 (+ 225us) server_negotiation.cc:407] Connection header received
0619 20:27:31.011688 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.011690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.011702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.011766 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:31.012767 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.013252 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.014258 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.014478 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.014562 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.016600 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:31.016602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:31.016603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:31.016610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:31.017773 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:31.018164 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:31.018165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:31.018166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:31.018225 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:31.018454 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:31.018455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:31.018464 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:31.018577 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:31.018661 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.018911 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.019000 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:27:31.019296 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.010823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39351 (local address 127.21.151.3:56391)
0619 20:27:31.010983 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.011024 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:31.011522 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:31.011854 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:31.011858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:31.012169 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:31.012680 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.012686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.013340 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.013341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.014138 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.014140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.014570 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.014571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.014687 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.015103 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:31.015112 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:31.016510 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:31.017868 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:31.017875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:31.017877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:31.018092 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:31.018288 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:31.018289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:31.018289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:31.018390 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:31.018652 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:31.018658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:31.018788 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 20:27:31.018889 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 20:27:31.019047 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260619 20:27:31.019757 26663 heartbeater.cc:347] Connected to a master server at 127.21.151.62:39351
I20260619 20:27:31.019848 26663 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:31.019963 26663 heartbeater.cc:511] Master 127.21.151.62:39351 requested a full tablet report, sending...
I20260619 20:27:31.020325 26207 ts_manager.cc:194] Registered new tserver with Master: 85b18525f34c4de68d94ae7169522968 (127.21.151.3:33709)
I20260619 20:27:31.020673 26207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:56391
I20260619 20:27:31.023027 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:27:31.030292 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.024324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.21.151.62:39351)
0619 20:27:31.024394 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.024396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.024481 (+ 85us) server_negotiation.cc:407] Connection header received
0619 20:27:31.024567 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.024568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.024577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.024625 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:31.025425 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.026029 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.026822 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.026996 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.027049 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.028117 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:31.028118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:31.028118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:31.028124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:31.029435 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:31.029702 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:31.029703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:31.029703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:31.029756 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:31.029903 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:31.029903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:31.029903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:31.030027 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:31.030098 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.030178 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.030231 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:27:31.031855 26207 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:31.033397 26207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:31.039462 26674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.035468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:36873 (local address 127.0.0.1:33942)
0619 20:27:31.035570 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.035610 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:31.035719 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:31.036055 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:31.036058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:31.036060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:31.036127 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.036129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.037295 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.037296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.038379 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.038381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.039149 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.039150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.039228 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.039241 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:27:31.039324 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 20:27:31.039379 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":38,"threads_started":1}
I20260619 20:27:31.039695 26676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.035682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:33709 (local address 127.0.0.1:38098)
0619 20:27:31.036155 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.036165 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:31.036211 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:31.036400 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:31.036401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:31.036403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:31.036440 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.036441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.037621 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.037622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.038523 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.038525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.039433 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.039434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.039491 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.039495 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:31.039549 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:27:31.039599 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":45,"thread_start_us":61,"threads_started":1}
I20260619 20:27:31.039978 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.035776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.21.151.3:33709)
0619 20:27:31.036141 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.036144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.036266 (+ 122us) server_negotiation.cc:407] Connection header received
0619 20:27:31.036309 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.036312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.036327 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.036402 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:31.036496 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.037499 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.038606 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.039364 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.039431 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.039495 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.039869 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.039917 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":87,"threads_started":1}
I20260619 20:27:31.040522 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.035647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.21.151.1:36873)
0619 20:27:31.035850 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.035856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.035868 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:27:31.035910 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.035912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.035928 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.036049 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:31.036198 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.037205 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.038460 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.039046 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.039146 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.039212 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.040390 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.040436 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
I20260619 20:27:31.040570 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.035830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:38183 (local address 127.0.0.1:39840)
0619 20:27:31.036252 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.036263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:31.036318 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:31.036609 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:31.036610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:31.036611 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:31.036651 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.036652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.037857 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.037858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.039274 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.039277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.040304 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.040305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.040351 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.040354 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:31.040445 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 20:27:31.040493 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":64,"threads_started":1}
I20260619 20:27:31.040835 26679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.035948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.21.151.2:38183)
0619 20:27:31.036096 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.036101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.036377 (+ 276us) server_negotiation.cc:407] Connection header received
0619 20:27:31.036423 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.036426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.036443 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.036526 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:31.036735 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.037761 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.039356 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.040224 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.040303 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.040353 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.040731 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.040773 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260619 20:27:31.040709 26597 tablet_service.cc:1467] Processing CreateTablet for tablet c40a68a307be4ad895da0275096560c9 (DEFAULT_TABLE table=test-table [id=d81ccddea57843608b1cd0d5b70d8e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:31.041097 26597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40a68a307be4ad895da0275096560c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:31.041407 26331 tablet_service.cc:1467] Processing CreateTablet for tablet c40a68a307be4ad895da0275096560c9 (DEFAULT_TABLE table=test-table [id=d81ccddea57843608b1cd0d5b70d8e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:31.041447 26464 tablet_service.cc:1467] Processing CreateTablet for tablet c40a68a307be4ad895da0275096560c9 (DEFAULT_TABLE table=test-table [id=d81ccddea57843608b1cd0d5b70d8e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:31.041738 26464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40a68a307be4ad895da0275096560c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:31.041738 26331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40a68a307be4ad895da0275096560c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:31.043963 26680 tablet_bootstrap.cc:492] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: Bootstrap starting.
I20260619 20:27:31.044548 26680 tablet_bootstrap.cc:654] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:31.044986 26682 tablet_bootstrap.cc:492] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d: Bootstrap starting.
I20260619 20:27:31.045164 26680 log.cc:824] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:31.045674 26682 tablet_bootstrap.cc:654] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:31.045961 26680 tablet_bootstrap.cc:492] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: No bootstrap required, opened a new log
I20260619 20:27:31.046065 26680 ts_tablet_manager.cc:1397] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:27:31.046278 26682 log.cc:824] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:31.046633 26681 tablet_bootstrap.cc:492] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: Bootstrap starting.
I20260619 20:27:31.046998 26682 tablet_bootstrap.cc:492] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d: No bootstrap required, opened a new log
I20260619 20:27:31.047088 26682 ts_tablet_manager.cc:1397] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:27:31.047312 26681 tablet_bootstrap.cc:654] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:31.047932 26681 log.cc:824] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:31.048326 26680 raft_consensus.cc:348] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } }
I20260619 20:27:31.048499 26680 raft_consensus.cc:374] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:31.048540 26680 raft_consensus.cc:729] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b18525f34c4de68d94ae7169522968, State: Initialized, Role: FOLLOWER
I20260619 20:27:31.048700 26681 tablet_bootstrap.cc:492] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: No bootstrap required, opened a new log
I20260619 20:27:31.048702 26680 consensus_queue.cc:260] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } }
I20260619 20:27:31.048774 26681 ts_tablet_manager.cc:1397] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:27:31.048883 26682 raft_consensus.cc:348] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } }
I20260619 20:27:31.049069 26663 heartbeater.cc:503] Master 127.21.151.62:39351 was elected leader, sending a full tablet report...
I20260619 20:27:31.049090 26682 raft_consensus.cc:374] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:31.049165 26682 raft_consensus.cc:729] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbac29b8f0e840b895d4547e80f1285d, State: Initialized, Role: FOLLOWER
I20260619 20:27:31.049278 26682 consensus_queue.cc:260] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } }
I20260619 20:27:31.049346 26680 ts_tablet_manager.cc:1428] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:27:31.049612 26397 heartbeater.cc:503] Master 127.21.151.62:39351 was elected leader, sending a full tablet report...
I20260619 20:27:31.049597 26682 ts_tablet_manager.cc:1428] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:27:31.050709 26681 raft_consensus.cc:348] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } }
I20260619 20:27:31.050833 26681 raft_consensus.cc:374] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:31.050854 26681 raft_consensus.cc:729] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35315045328494eb30b9ed0feb20980, State: Initialized, Role: FOLLOWER
I20260619 20:27:31.050981 26681 consensus_queue.cc:260] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } }
I20260619 20:27:31.051239 26681 ts_tablet_manager.cc:1428] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:27:31.051258 26530 heartbeater.cc:503] Master 127.21.151.62:39351 was elected leader, sending a full tablet report...
W20260619 20:27:31.104439 26531 tablet.cc:2369] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:27:31.184962 26398 tablet.cc:2369] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:27:31.260701 26664 tablet.cc:2369] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:27:31.381453 26687 raft_consensus.cc:482] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:31.381585 26687 raft_consensus.cc:504] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } }
I20260619 20:27:31.381892 26687 leader_election.cc:288] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85b18525f34c4de68d94ae7169522968 (127.21.151.3:33709), d35315045328494eb30b9ed0feb20980 (127.21.151.2:38183)
I20260619 20:27:31.384967 26688 raft_consensus.cc:482] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:31.385040 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.382175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:44783 (local address 127.21.151.3:33709)
0619 20:27:31.382233 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.382235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.382381 (+ 146us) server_negotiation.cc:407] Connection header received
0619 20:27:31.382415 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.382416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.382427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.382534 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:31.382693 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.383221 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.384247 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.384696 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.384784 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.384793 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.384940 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.384973 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:27:31.385066 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.382112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:33709 (local address 127.21.151.1:44783)
0619 20:27:31.382232 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.382243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:31.382326 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:31.382544 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:31.382545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:31.382548 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:31.382618 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.382620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.383309 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.383310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.384160 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.384162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.384785 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.384786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.384854 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.384858 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:31.384933 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:27:31.384973 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":10,"thread_start_us":79,"threads_started":1}
I20260619 20:27:31.385151 26679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.382394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:39157 (local address 127.21.151.2:38183)
0619 20:27:31.382449 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.382452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.382461 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:31.382535 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.382536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.382546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.382590 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:31.382738 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.383239 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.384324 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.384805 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.384890 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.384897 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.385052 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.385092 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 20:27:31.385321 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.382320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:38183 (local address 127.21.151.1:39157)
0619 20:27:31.382416 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.382426 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:31.382476 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:31.382602 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:31.382602 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:31.382604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:31.382677 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.382678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.383317 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.383318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.384213 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.384216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.384889 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.384890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.384963 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.384966 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:31.385191 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 20:27:31.385238 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":41,"threads_started":1}
I20260619 20:27:31.385337 26688 raft_consensus.cc:504] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } }
I20260619 20:27:31.385515 26617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40a68a307be4ad895da0275096560c9" candidate_uuid: "fbac29b8f0e840b895d4547e80f1285d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b18525f34c4de68d94ae7169522968" is_pre_election: true
I20260619 20:27:31.385685 26617 raft_consensus.cc:2454] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbac29b8f0e840b895d4547e80f1285d in term 0.
I20260619 20:27:31.385788 26688 leader_election.cc:288] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbac29b8f0e840b895d4547e80f1285d (127.21.151.1:36873), 85b18525f34c4de68d94ae7169522968 (127.21.151.3:33709)
I20260619 20:27:31.385869 26484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40a68a307be4ad895da0275096560c9" candidate_uuid: "fbac29b8f0e840b895d4547e80f1285d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35315045328494eb30b9ed0feb20980" is_pre_election: true
I20260619 20:27:31.385970 26484 raft_consensus.cc:2454] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbac29b8f0e840b895d4547e80f1285d in term 0.
I20260619 20:27:31.386160 26284 leader_election.cc:302] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d35315045328494eb30b9ed0feb20980, fbac29b8f0e840b895d4547e80f1285d; no voters:
I20260619 20:27:31.386358 26687 raft_consensus.cc:2781] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:31.386442 26687 raft_consensus.cc:482] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:31.386487 26687 raft_consensus.cc:3037] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:31.387428 26687 raft_consensus.cc:504] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } }
I20260619 20:27:31.387558 26687 leader_election.cc:288] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [CANDIDATE]: Term 1 election: Requested vote from peers 85b18525f34c4de68d94ae7169522968 (127.21.151.3:33709), d35315045328494eb30b9ed0feb20980 (127.21.151.2:38183)
I20260619 20:27:31.387691 26617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40a68a307be4ad895da0275096560c9" candidate_uuid: "fbac29b8f0e840b895d4547e80f1285d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b18525f34c4de68d94ae7169522968"
I20260619 20:27:31.387758 26617 raft_consensus.cc:3037] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:31.387759 26484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40a68a307be4ad895da0275096560c9" candidate_uuid: "fbac29b8f0e840b895d4547e80f1285d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35315045328494eb30b9ed0feb20980"
I20260619 20:27:31.387822 26484 raft_consensus.cc:3037] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:31.388667 26617 raft_consensus.cc:2454] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbac29b8f0e840b895d4547e80f1285d in term 1.
I20260619 20:27:31.388707 26484 raft_consensus.cc:2454] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbac29b8f0e840b895d4547e80f1285d in term 1.
I20260619 20:27:31.388870 26284 leader_election.cc:302] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d35315045328494eb30b9ed0feb20980, fbac29b8f0e840b895d4547e80f1285d; no voters:
I20260619 20:27:31.388998 26687 raft_consensus.cc:2781] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:31.389194 26687 raft_consensus.cc:686] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [term 1 LEADER]: Becoming Leader. State: Replica: fbac29b8f0e840b895d4547e80f1285d, State: Running, Role: LEADER
I20260619 20:27:31.389300 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.386191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:41071 (local address 127.21.151.3:33709)
0619 20:27:31.386225 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.386227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.386265 (+ 38us) server_negotiation.cc:407] Connection header received
0619 20:27:31.386321 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.386321 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.386330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.386378 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:31.386547 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.387035 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.388413 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.388937 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.388982 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.388992 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.389213 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.389250 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:27:31.389523 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.386065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:33709 (local address 127.21.151.2:41071)
0619 20:27:31.386207 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.386218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:31.386274 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:31.386414 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:31.386415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:31.386417 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:31.386482 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.386483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.387111 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.387112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.388310 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.388312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.389052 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.389053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.389127 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.389131 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:31.389392 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 20:27:31.389432 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":50,"threads_started":1}
I20260619 20:27:31.389530 26687 consensus_queue.cc:237] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } }
I20260619 20:27:31.389719 26617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40a68a307be4ad895da0275096560c9" candidate_uuid: "d35315045328494eb30b9ed0feb20980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b18525f34c4de68d94ae7169522968" is_pre_election: true
I20260619 20:27:31.389807 26617 raft_consensus.cc:2379] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d35315045328494eb30b9ed0feb20980 in current term 1: Already voted for candidate fbac29b8f0e840b895d4547e80f1285d in this term.
I20260619 20:27:31.389936 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.385907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:36873 (local address 127.21.151.2:41733)
0619 20:27:31.386021 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.386031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:31.386077 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:31.386259 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:31.386260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:31.386262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:31.386320 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.386321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.387673 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.387674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.388924 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:31.388926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.389696 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.389697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:31.389771 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.389775 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:31.389821 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 20:27:31.389856 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260619 20:27:31.390133 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.385952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:41733 (local address 127.21.151.1:36873)
0619 20:27:31.386066 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.386068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.386078 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:31.386158 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.386159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.386168 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.386197 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:31.386406 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.387028 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.389023 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.389590 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.389652 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.389659 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.390041 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.390079 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 20:27:31.390327 26351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c40a68a307be4ad895da0275096560c9" candidate_uuid: "d35315045328494eb30b9ed0feb20980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbac29b8f0e840b895d4547e80f1285d" is_pre_election: true
I20260619 20:27:31.390691 26417 leader_election.cc:302] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d35315045328494eb30b9ed0feb20980; no voters: 85b18525f34c4de68d94ae7169522968, fbac29b8f0e840b895d4547e80f1285d
I20260619 20:27:31.390878 26688 raft_consensus.cc:2726] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 20:27:31.390880 26206 catalog_manager.cc:5495] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d reported cstate change: term changed from 0 to 1, leader changed from <none> to fbac29b8f0e840b895d4547e80f1285d (127.21.151.1). New cstate: current_term: 1 leader_uuid: "fbac29b8f0e840b895d4547e80f1285d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbac29b8f0e840b895d4547e80f1285d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 } health_report { overall_health: UNKNOWN } } }
I20260619 20:27:31.478902 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.476785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.21.151.1:36873)
0619 20:27:31.476837 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.476839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.476846 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:27:31.476903 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.476904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.476921 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.476996 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:31.477111 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.477576 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.478291 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:31.478472 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:31.478523 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:31.478641 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:31.478741 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:31.478796 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.478829 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.478844 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:27:31.480693 26617 raft_consensus.cc:1261] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [term 1 FOLLOWER]: Refusing update from remote peer fbac29b8f0e840b895d4547e80f1285d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:31.480731 26484 raft_consensus.cc:1261] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 1 FOLLOWER]: Refusing update from remote peer fbac29b8f0e840b895d4547e80f1285d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:31.481055 26687 consensus_queue.cc:1035] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b18525f34c4de68d94ae7169522968" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 33709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:31.481220 26693 consensus_queue.cc:1035] T c40a68a307be4ad895da0275096560c9 P fbac29b8f0e840b895d4547e80f1285d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35315045328494eb30b9ed0feb20980" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:31.483544 26698 mvcc.cc:205] Tried to move back new op lower bound from 7298665887662415872 to 7298665887293313024. Current Snapshot: MvccSnapshot[applied={T|T < 7298665887662415872}]
I20260619 20:27:31.484002 26696 mvcc.cc:205] Tried to move back new op lower bound from 7298665887662415872 to 7298665887293313024. Current Snapshot: MvccSnapshot[applied={T|T < 7298665887662415872}]
I20260619 20:27:31.486023 26207 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:31.486117 26207 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:31.486855 26207 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6c1886d01f640e4a5a4022cada36dd4: Sending DeleteTablet for 3 replicas of tablet c40a68a307be4ad895da0275096560c9
I20260619 20:27:31.487205 26597 tablet_service.cc:1514] Processing DeleteTablet for tablet c40a68a307be4ad895da0275096560c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:31 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:38098
I20260619 20:27:31.487238 26464 tablet_service.cc:1514] Processing DeleteTablet for tablet c40a68a307be4ad895da0275096560c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:31 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:39840
I20260619 20:27:31.487408 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 26269
I20260619 20:27:31.487440 26331 tablet_service.cc:1514] Processing DeleteTablet for tablet c40a68a307be4ad895da0275096560c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:31 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:33942
I20260619 20:27:31.488166 26706 tablet_replica.cc:330] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: stopping tablet replica
I20260619 20:27:31.488265 26705 tablet_replica.cc:330] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: stopping tablet replica
I20260619 20:27:31.488267 26706 raft_consensus.cc:2229] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:31.488339 26705 raft_consensus.cc:2229] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:31.488366 26706 raft_consensus.cc:2258] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:31.488417 26705 raft_consensus.cc:2258] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:31.488842 26706 ts_tablet_manager.cc:1905] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:31.488883 26705 ts_tablet_manager.cc:1905] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:31.490314 26706 ts_tablet_manager.cc:1918] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:31.490363 26706 log.cc:1196] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/c40a68a307be4ad895da0275096560c9
I20260619 20:27:31.490618 26706 ts_tablet_manager.cc:1939] T c40a68a307be4ad895da0275096560c9 P d35315045328494eb30b9ed0feb20980: Deleting consensus metadata
I20260619 20:27:31.490772 26705 ts_tablet_manager.cc:1918] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:31.490834 26705 log.cc:1196] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/c40a68a307be4ad895da0275096560c9
I20260619 20:27:31.490954 26191 catalog_manager.cc:4841] TS d35315045328494eb30b9ed0feb20980 (127.21.151.2:38183): tablet c40a68a307be4ad895da0275096560c9 (table test-table [id=d81ccddea57843608b1cd0d5b70d8e4c]) successfully deleted
I20260619 20:27:31.491061 26705 ts_tablet_manager.cc:1939] T c40a68a307be4ad895da0275096560c9 P 85b18525f34c4de68d94ae7169522968: Deleting consensus metadata
I20260619 20:27:31.491448 26191 catalog_manager.cc:4841] TS 85b18525f34c4de68d94ae7169522968 (127.21.151.3:33709): tablet c40a68a307be4ad895da0275096560c9 (table test-table [id=d81ccddea57843608b1cd0d5b70d8e4c]) successfully deleted
W20260619 20:27:31.495154 26191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:36873 (error 108)
I20260619 20:27:31.495703 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.495558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:36873 (local address 127.0.0.1:33952)
0619 20:27:31.495640 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.495679 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 20:27:31.495749 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 26402
W20260619 20:27:31.496659 26191 catalog_manager.cc:4584] TS fbac29b8f0e840b895d4547e80f1285d (127.21.151.1:36873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c40a68a307be4ad895da0275096560c9: Network error: Client connection negotiation failed: client connection to 127.21.151.1:36873: connect: Connection refused (error 111)
I20260619 20:27:31.500638 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 26535
I20260619 20:27:31.504748 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 26176
2026-06-19T20:27:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2521 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T20:27:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:31Z Disabled control of system clock
I20260619 20:27:31.531729 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44153
--webserver_interface=127.21.151.62
--webserver_port=0
--builtin_ntp_servers=127.21.151.20:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:44153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 20:27:31.617668 26711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:31.617883 26711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:31.617908 26711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:31.619045 26711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:31.619112 26711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:31.619128 26711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:31.619156 26711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:31.619171 26711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:31.620465 26711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:44153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:31.620707 26711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:31.620967 26711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:31.625823 26717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:31.625862 26716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:31.625918 26711 server_base.cc:1034] running on GCE node
W20260619 20:27:31.626042 26719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:31.626324 26711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:31.626619 26711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:31.627780 26711 hybrid_clock.cc:648] HybridClock initialized: now 1781900851627767 us; error 27 us; skew 500 ppm
I20260619 20:27:31.629127 26711 webserver.cc:460] Webserver started at http://127.21.151.62:39215/ using document root <none> and password file <none>
I20260619 20:27:31.629379 26711 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:31.629467 26711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:31.629621 26711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:31.630674 26711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "3afbc84d699d4de792ad18ef55a698a5"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "51644ada1222c7199af37c42bbff236f"
server_key_iv: "f0e32c74edebff110ec6715eb996308d"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.631057 26711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "3afbc84d699d4de792ad18ef55a698a5"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "51644ada1222c7199af37c42bbff236f"
server_key_iv: "f0e32c74edebff110ec6715eb996308d"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.632441 26711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:31.633172 26725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.633332 26711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:31.633440 26711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "3afbc84d699d4de792ad18ef55a698a5"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "51644ada1222c7199af37c42bbff236f"
server_key_iv: "f0e32c74edebff110ec6715eb996308d"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.633513 26711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:31.637171 26711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:31.637426 26711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:31.637564 26711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:31.641336 26711 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:44153
I20260619 20:27:31.641366 26777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:44153 every 8 connection(s)
I20260619 20:27:31.641687 26711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:31.642272 26778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:31.644896 26778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5: Bootstrap starting.
I20260619 20:27:31.645443 26778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:31.645732 26778 log.cc:824] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:31.646386 26778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5: No bootstrap required, opened a new log
I20260619 20:27:31.647964 26778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afbc84d699d4de792ad18ef55a698a5" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44153 } }
I20260619 20:27:31.648113 26778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:31.648156 26778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afbc84d699d4de792ad18ef55a698a5, State: Initialized, Role: FOLLOWER
I20260619 20:27:31.648294 26778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afbc84d699d4de792ad18ef55a698a5" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44153 } }
I20260619 20:27:31.648353 26778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:31.648391 26778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:31.648442 26778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:31.648567 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 26711
I20260619 20:27:31.648667 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:31.648835 22108 external_mini_cluster.cc:1428] Setting key 7b4e60f03808ed33b0d9566891d50945
I20260619 20:27:31.649159 26778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afbc84d699d4de792ad18ef55a698a5" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44153 } }
I20260619 20:27:31.649263 26778 leader_election.cc:302] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3afbc84d699d4de792ad18ef55a698a5; no voters:
I20260619 20:27:31.649422 26778 leader_election.cc:288] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:31.649569 26783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:31.649681 26778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:31.649730 26783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3afbc84d699d4de792ad18ef55a698a5, State: Running, Role: LEADER
I20260619 20:27:31.650177 26783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afbc84d699d4de792ad18ef55a698a5" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44153 } }
I20260619 20:27:31.650969 26784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3afbc84d699d4de792ad18ef55a698a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afbc84d699d4de792ad18ef55a698a5" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44153 } } }
I20260619 20:27:31.651065 26784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:31.651199 26784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3afbc84d699d4de792ad18ef55a698a5. Latest consensus state: current_term: 1 leader_uuid: "3afbc84d699d4de792ad18ef55a698a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afbc84d699d4de792ad18ef55a698a5" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44153 } } }
I20260619 20:27:31.651245 26784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:31.651511 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.649297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54346 (local address 127.21.151.62:44153)
0619 20:27:31.649484 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.649493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.649536 (+ 43us) server_negotiation.cc:407] Connection header received
0619 20:27:31.649646 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.649653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.649696 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.649818 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:31.650548 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:31.650552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:31.650552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:31.650557 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:31.650598 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:31.650611 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:31.650685 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.650921 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.650975 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260619 20:27:31.651286 26796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:31.652182 26796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:31.653558 26796 catalog_manager.cc:1347] Generated new cluster ID: d7b49cea63cb4cc68b68602967e8d847
I20260619 20:27:31.653606 26796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:31.659850 26796 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:31.660007 26796 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:31.667469 26796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5: Generated new TSK 0
I20260619 20:27:31.667675 26796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:27:31.669744 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44153
--builtin_ntp_servers=127.21.151.20:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 20:27:31.747529 26802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:31.747792 26802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:31.747833 26802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:31.748955 26802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:31.749013 26802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:31.749053 26802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:31.750525 26802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:31.750794 26802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:31.751083 26802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:31.756009 26810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:31.756012 26807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:31.756027 26808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:31.756399 26802 server_base.cc:1034] running on GCE node
I20260619 20:27:31.756592 26802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:31.756865 26802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:31.758049 26802 hybrid_clock.cc:648] HybridClock initialized: now 1781900851758027 us; error 32 us; skew 500 ppm
I20260619 20:27:31.759552 26802 webserver.cc:460] Webserver started at http://127.21.151.1:40783/ using document root <none> and password file <none>
I20260619 20:27:31.759860 26802 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:31.759936 26802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:31.760079 26802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:31.761507 26802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "09233a09a00e42de8081a12cd3be0b01"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "162b69b54f26c91264a78feb9194ff40"
server_key_iv: "8f678d82c1fe7fc1131c5f5d8fd10018"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.761946 26802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "09233a09a00e42de8081a12cd3be0b01"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "162b69b54f26c91264a78feb9194ff40"
server_key_iv: "8f678d82c1fe7fc1131c5f5d8fd10018"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.763780 26802 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 20:27:31.764740 26816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.764957 26802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:31.765057 26802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "09233a09a00e42de8081a12cd3be0b01"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "162b69b54f26c91264a78feb9194ff40"
server_key_iv: "8f678d82c1fe7fc1131c5f5d8fd10018"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.765138 26802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:31.770016 26802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:31.770291 26802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:31.770409 26802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:31.770605 26802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:31.770896 26802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:31.770933 26802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.770959 26802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:31.770977 26802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.777568 26802 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:43169
I20260619 20:27:31.777601 26929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:43169 every 8 connection(s)
I20260619 20:27:31.777951 26802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:27:31.781337 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.778675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:56411 (local address 127.21.151.62:44153)
0619 20:27:31.778748 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.778751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.779080 (+ 329us) server_negotiation.cc:407] Connection header received
0619 20:27:31.779668 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.779669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.779677 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.779731 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:31.781084 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:31.781086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:31.781086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:31.781092 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:31.781107 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:31.781111 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:31.781165 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.781247 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.781264 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:31.781608 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.778677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44153 (local address 127.21.151.1:56411)
0619 20:27:31.778925 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.778957 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:31.779533 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:31.779807 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:31.779810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:31.779818 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:31.780931 (+ 1113us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:27:31.780931 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:31.780959 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:27:31.780988 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:31.781160 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:31.781165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:31.781170 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:27:31.781258 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 20:27:31.781320 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
I20260619 20:27:31.782073 26930 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44153
I20260619 20:27:31.782174 26930 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:31.782326 26930 heartbeater.cc:511] Master 127.21.151.62:44153 requested a full tablet report, sending...
I20260619 20:27:31.782804 26742 ts_manager.cc:194] Registered new tserver with Master: 09233a09a00e42de8081a12cd3be0b01 (127.21.151.1:43169)
I20260619 20:27:31.784482 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 26802
I20260619 20:27:31.784581 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:31.784742 22108 external_mini_cluster.cc:1428] Setting key 3c01439f650ce3384e8da5c1bbbed56a
I20260619 20:27:31.785601 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44153
--builtin_ntp_servers=127.21.151.20:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 20:27:31.864809 26933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:31.865027 26933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:31.865063 26933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:31.866187 26933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:31.866245 26933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:31.866299 26933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:31.867584 26933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:31.867843 26933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:31.868113 26933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:31.873195 26939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:31.873186 26938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:31.873314 26941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:31.873457 26933 server_base.cc:1034] running on GCE node
I20260619 20:27:31.873703 26933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:31.873955 26933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:31.875108 26933 hybrid_clock.cc:648] HybridClock initialized: now 1781900851875095 us; error 25 us; skew 500 ppm
I20260619 20:27:31.876524 26933 webserver.cc:460] Webserver started at http://127.21.151.2:40265/ using document root <none> and password file <none>
I20260619 20:27:31.876777 26933 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:31.876849 26933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:31.876996 26933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:31.878065 26933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "d61bbcd2673e49509cdd5827733ccf1c"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "238ca6c78263e9a4077e97f98baaf512"
server_key_iv: "b3bdc7c8a925f707c38921af46a80d11"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.878443 26933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "d61bbcd2673e49509cdd5827733ccf1c"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "238ca6c78263e9a4077e97f98baaf512"
server_key_iv: "b3bdc7c8a925f707c38921af46a80d11"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.879750 26933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:31.880435 26947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.880599 26933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:31.880707 26933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "d61bbcd2673e49509cdd5827733ccf1c"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "238ca6c78263e9a4077e97f98baaf512"
server_key_iv: "b3bdc7c8a925f707c38921af46a80d11"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.880788 26933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:31.884083 26933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:31.884333 26933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:31.884477 26933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:31.884688 26933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:31.884979 26933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:31.885035 26933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.885074 26933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:31.885104 26933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.891008 26933 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:35061
I20260619 20:27:31.891041 27060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:35061 every 8 connection(s)
I20260619 20:27:31.891366 26933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:27:31.894068 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.892127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:46181 (local address 127.21.151.62:44153)
0619 20:27:31.892188 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:31.892191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:31.892422 (+ 231us) server_negotiation.cc:407] Connection header received
0619 20:27:31.892852 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:31.892853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:31.892858 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:31.892897 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:31.893891 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:31.893892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:31.893892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:31.893895 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:31.893908 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:31.893912 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:31.893953 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:31.894009 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 20:27:31.894020 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:27:31.894258 27063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:31.892121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44153 (local address 127.21.151.2:46181)
0619 20:27:31.892304 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:31.892336 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:31.892778 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:31.892949 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:31.892952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:31.892959 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:31.893766 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:27:31.893766 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:31.893791 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:27:31.893816 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:31.893952 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:31.893955 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:31.893957 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:27:31.894008 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:27:31.894049 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260619 20:27:31.894714 27061 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44153
I20260619 20:27:31.894798 27061 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:31.894943 27061 heartbeater.cc:511] Master 127.21.151.62:44153 requested a full tablet report, sending...
I20260619 20:27:31.895332 26742 ts_manager.cc:194] Registered new tserver with Master: d61bbcd2673e49509cdd5827733ccf1c (127.21.151.2:35061)
I20260619 20:27:31.900094 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 26933
I20260619 20:27:31.900167 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:31.900375 22108 external_mini_cluster.cc:1428] Setting key 09a68ceda849c38e2d54bdd3a180df38
I20260619 20:27:31.901238 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44153
--builtin_ntp_servers=127.21.151.20:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 20:27:31.979316 27064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:31.979542 27064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:31.979576 27064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:31.980751 27064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:31.980815 27064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:31.980855 27064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:31.982162 27064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:31.982410 27064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:31.982699 27064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:31.987479 27069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:31.987532 27070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:31.987671 27064 server_base.cc:1034] running on GCE node
W20260619 20:27:31.987705 27072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:31.987890 27064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:31.988108 27064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:31.989274 27064 hybrid_clock.cc:648] HybridClock initialized: now 1781900851989271 us; error 35 us; skew 500 ppm
I20260619 20:27:31.990531 27064 webserver.cc:460] Webserver started at http://127.21.151.3:37261/ using document root <none> and password file <none>
I20260619 20:27:31.990787 27064 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:31.990854 27064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:31.990984 27064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:31.992108 27064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "2644b5c454124816b2fa5484319aa47a"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "63c27d294ca7a1ce857d86e2db636f94"
server_key_iv: "ff6c10a7dffdc33f20563daad2a90611"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.992449 27064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "2644b5c454124816b2fa5484319aa47a"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "63c27d294ca7a1ce857d86e2db636f94"
server_key_iv: "ff6c10a7dffdc33f20563daad2a90611"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.993657 27064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:31.994295 27078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.994446 27064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:31.994545 27064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "2644b5c454124816b2fa5484319aa47a"
format_stamp: "Formatted at 2026-06-19 20:27:31 on dist-test-slave-f9rk"
server_key: "63c27d294ca7a1ce857d86e2db636f94"
server_key_iv: "ff6c10a7dffdc33f20563daad2a90611"
server_key_version: "encryptionkey@0"
I20260619 20:27:31.994611 27064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:31.997905 27064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:31.998147 27064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:31.998283 27064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:31.998468 27064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:31.998756 27064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:31.998805 27064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:31.998843 27064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:31.998870 27064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:32.004403 27064 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:41263
I20260619 20:27:32.004441 27191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:41263 every 8 connection(s)
I20260619 20:27:32.004765 27064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:32.005533 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 27064
I20260619 20:27:32.005633 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:32.005820 22108 external_mini_cluster.cc:1428] Setting key 49e85703668d8be4af57acc8f14945be
I20260619 20:27:32.007706 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.005380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:43695 (local address 127.21.151.62:44153)
0619 20:27:32.005454 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.005457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.005762 (+ 305us) server_negotiation.cc:407] Connection header received
0619 20:27:32.006312 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.006313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.006320 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.006373 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.007473 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.007474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.007475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.007479 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.007493 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.007497 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.007543 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.007586 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.007597 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:27:32.007936 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.005379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44153 (local address 127.21.151.3:43695)
0619 20:27:32.005613 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.005660 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:32.006191 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:32.006444 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:32.006447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:32.006457 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:32.007342 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:27:32.007343 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:32.007368 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:27:32.007394 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:32.007540 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:32.007542 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:32.007544 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:27:32.007592 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:27:32.007689 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":121,"threads_started":1}
I20260619 20:27:32.008322 27192 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44153
I20260619 20:27:32.008420 27192 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:32.008561 27192 heartbeater.cc:511] Master 127.21.151.62:44153 requested a full tablet report, sending...
I20260619 20:27:32.008885 26742 ts_manager.cc:194] Registered new tserver with Master: 2644b5c454124816b2fa5484319aa47a (127.21.151.3:41263)
I20260619 20:27:32.017346 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:27:32.019692 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.018612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.21.151.62:44153)
0619 20:27:32.018656 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.018658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.018779 (+ 121us) server_negotiation.cc:407] Connection header received
0619 20:27:32.018823 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.018824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.018828 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.018876 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.019497 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.019498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.019498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.019506 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.019516 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.019519 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.019556 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.019588 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.019600 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:27:32.021035 26742 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:32.022509 26742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:32.026247 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.024533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:41263 (local address 127.0.0.1:41540)
0619 20:27:32.024689 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.024727 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:32.024986 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:32.025396 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:32.025398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:32.025400 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:32.025807 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:27:32.025808 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:32.025826 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:27:32.025850 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:32.026063 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:32.026069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:32.026072 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:32.026143 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 20:27:32.026155 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":56,"threads_started":1}
I20260619 20:27:32.026268 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.024610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.21.151.3:41263)
0619 20:27:32.025257 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.025262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.025270 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:32.025293 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.025296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.025303 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.025541 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.025919 (+ 378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.025924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.025924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.025928 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.025961 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.025978 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.026065 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.026163 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.026180 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":81,"threads_started":1}
I20260619 20:27:32.026506 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.024566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:35061 (local address 127.0.0.1:37804)
0619 20:27:32.025074 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.025081 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:32.025129 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:32.025625 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:32.025626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:32.025627 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:32.026103 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:27:32.026103 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:32.026108 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:27:32.026135 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:32.026380 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:32.026380 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:32.026381 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:27:32.026426 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 20:27:32.026438 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":12,"thread_start_us":80,"threads_started":1}
I20260619 20:27:32.026727 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.024566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:43169 (local address 127.0.0.1:45228)
0619 20:27:32.024985 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.024995 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:32.025064 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:32.025217 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:32.025223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:32.025230 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:32.026354 (+ 1124us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:27:32.026354 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:32.026360 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:27:32.026372 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:32.026608 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:32.026609 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:32.026609 (+ 0us) client_negotiation.cc:769] Sending connection context
0619 20:27:32.026650 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 20:27:32.026662 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":83,"threads_started":1}
I20260619 20:27:32.026777 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.024782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.21.151.2:35061)
0619 20:27:32.024934 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.024939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.025130 (+ 191us) server_negotiation.cc:407] Connection header received
0619 20:27:32.025191 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.025194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.025205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.025281 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.026216 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.026223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.026224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.026228 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.026262 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.026289 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.026352 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.026680 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.026697 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260619 20:27:32.026983 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.024784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.21.151.1:43169)
0619 20:27:32.024943 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.024949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.025060 (+ 111us) server_negotiation.cc:407] Connection header received
0619 20:27:32.025128 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.025131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.025141 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.025211 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.026439 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.026451 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.026451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.026456 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.026494 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.026527 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.026583 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.026854 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.026874 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260619 20:27:32.027108 27126 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea76ca960d0476ea84b02e6a354922a (DEFAULT_TABLE table=test-table [id=2f61c5ddea3d4774bf6189ed6815a180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:32.027444 27126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea76ca960d0476ea84b02e6a354922a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:32.027522 26995 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea76ca960d0476ea84b02e6a354922a (DEFAULT_TABLE table=test-table [id=2f61c5ddea3d4774bf6189ed6815a180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:32.027587 26864 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea76ca960d0476ea84b02e6a354922a (DEFAULT_TABLE table=test-table [id=2f61c5ddea3d4774bf6189ed6815a180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:32.027875 26995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea76ca960d0476ea84b02e6a354922a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:32.027884 26864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea76ca960d0476ea84b02e6a354922a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:32.029990 27206 tablet_bootstrap.cc:492] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: Bootstrap starting.
I20260619 20:27:32.030539 27206 tablet_bootstrap.cc:654] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:32.030755 27206 log.cc:824] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:32.031009 27207 tablet_bootstrap.cc:492] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01: Bootstrap starting.
I20260619 20:27:32.031678 27207 tablet_bootstrap.cc:654] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:32.031780 27208 tablet_bootstrap.cc:492] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: Bootstrap starting.
I20260619 20:27:32.031986 27206 tablet_bootstrap.cc:492] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: No bootstrap required, opened a new log
I20260619 20:27:32.032006 27207 log.cc:824] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:32.032058 27206 ts_tablet_manager.cc:1397] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:32.032402 27208 tablet_bootstrap.cc:654] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:32.032673 27208 log.cc:824] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:32.032744 27207 tablet_bootstrap.cc:492] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01: No bootstrap required, opened a new log
I20260619 20:27:32.032817 27207 ts_tablet_manager.cc:1397] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:27:32.033288 27208 tablet_bootstrap.cc:492] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: No bootstrap required, opened a new log
I20260619 20:27:32.033363 27208 ts_tablet_manager.cc:1397] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:32.034641 27206 raft_consensus.cc:348] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 } } peers { permanent_uuid: "09233a09a00e42de8081a12cd3be0b01" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 43169 } } peers { permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 } }
I20260619 20:27:32.034845 27206 raft_consensus.cc:374] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:32.034885 27206 raft_consensus.cc:729] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2644b5c454124816b2fa5484319aa47a, State: Initialized, Role: FOLLOWER
I20260619 20:27:32.035018 27206 consensus_queue.cc:260] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 } } peers { permanent_uuid: "09233a09a00e42de8081a12cd3be0b01" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 43169 } } peers { permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 } }
I20260619 20:27:32.035280 27206 ts_tablet_manager.cc:1428] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 20:27:32.035308 27192 heartbeater.cc:503] Master 127.21.151.62:44153 was elected leader, sending a full tablet report...
I20260619 20:27:32.035444 27208 raft_consensus.cc:348] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 } } peers { permanent_uuid: "09233a09a00e42de8081a12cd3be0b01" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 43169 } } peers { permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 } }
I20260619 20:27:32.035605 27208 raft_consensus.cc:374] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:32.035669 27208 raft_consensus.cc:729] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61bbcd2673e49509cdd5827733ccf1c, State: Initialized, Role: FOLLOWER
I20260619 20:27:32.035674 27207 raft_consensus.cc:348] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 } } peers { permanent_uuid: "09233a09a00e42de8081a12cd3be0b01" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 43169 } } peers { permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 } }
I20260619 20:27:32.035784 27208 consensus_queue.cc:260] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 } } peers { permanent_uuid: "09233a09a00e42de8081a12cd3be0b01" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 43169 } } peers { permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 } }
I20260619 20:27:32.035825 27207 raft_consensus.cc:374] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:32.035866 27207 raft_consensus.cc:729] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09233a09a00e42de8081a12cd3be0b01, State: Initialized, Role: FOLLOWER
I20260619 20:27:32.035981 27207 consensus_queue.cc:260] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 } } peers { permanent_uuid: "09233a09a00e42de8081a12cd3be0b01" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 43169 } } peers { permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 } }
I20260619 20:27:32.036039 27208 ts_tablet_manager.cc:1428] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:27:32.036156 27061 heartbeater.cc:503] Master 127.21.151.62:44153 was elected leader, sending a full tablet report...
I20260619 20:27:32.036254 27207 ts_tablet_manager.cc:1428] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 20:27:32.036365 26930 heartbeater.cc:503] Master 127.21.151.62:44153 was elected leader, sending a full tablet report...
I20260619 20:27:32.057497 27214 raft_consensus.cc:482] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:32.057636 27214 raft_consensus.cc:504] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 } } peers { permanent_uuid: "09233a09a00e42de8081a12cd3be0b01" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 43169 } } peers { permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 } }
I20260619 20:27:32.057989 27214 leader_election.cc:288] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2644b5c454124816b2fa5484319aa47a (127.21.151.3:41263), d61bbcd2673e49509cdd5827733ccf1c (127.21.151.2:35061)
I20260619 20:27:32.059147 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.058129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:41263 (local address 127.21.151.1:50991)
0619 20:27:32.058167 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.058179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:32.058220 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:32.058374 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:32.058375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:32.058378 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:32.058811 (+ 433us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:27:32.058812 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:32.058819 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:27:32.058829 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:32.059006 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:32.059007 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:32.059008 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:27:32.059057 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:27:32.059071 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 20:27:32.059178 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.058204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:50991 (local address 127.21.151.3:41263)
0619 20:27:32.058258 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.058260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.058267 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:27:32.058291 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.058292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.058297 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.058356 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.058903 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.058904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.058904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.058908 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.058923 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.058928 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.059006 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.059082 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.059097 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:27:32.059283 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.058268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:36055 (local address 127.21.151.2:35061)
0619 20:27:32.058311 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.058318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.058358 (+ 40us) server_negotiation.cc:407] Connection header received
0619 20:27:32.058409 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.058410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.058415 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.058486 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.059080 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.059081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.059081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.059085 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.059099 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.059103 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.059151 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.059193 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.059206 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:32.059453 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.058150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:35061 (local address 127.21.151.1:36055)
0619 20:27:32.058312 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.058321 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:32.058362 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:32.058490 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:32.058491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:32.058492 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:32.058975 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:27:32.058975 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:32.058982 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:27:32.058994 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:32.059149 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:32.059149 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:32.059150 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:27:32.059377 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 20:27:32.059390 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260619 20:27:32.059438 27146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea76ca960d0476ea84b02e6a354922a" candidate_uuid: "09233a09a00e42de8081a12cd3be0b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2644b5c454124816b2fa5484319aa47a" is_pre_election: true
I20260619 20:27:32.059579 27146 raft_consensus.cc:2454] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09233a09a00e42de8081a12cd3be0b01 in term 0.
I20260619 20:27:32.059736 27015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea76ca960d0476ea84b02e6a354922a" candidate_uuid: "09233a09a00e42de8081a12cd3be0b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d61bbcd2673e49509cdd5827733ccf1c" is_pre_election: true
I20260619 20:27:32.059815 26817 leader_election.cc:302] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09233a09a00e42de8081a12cd3be0b01, 2644b5c454124816b2fa5484319aa47a; no voters:
I20260619 20:27:32.059883 27015 raft_consensus.cc:2454] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09233a09a00e42de8081a12cd3be0b01 in term 0.
I20260619 20:27:32.059919 27214 raft_consensus.cc:2781] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:32.059989 27214 raft_consensus.cc:482] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:32.060020 27214 raft_consensus.cc:3037] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:32.060746 27214 raft_consensus.cc:504] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 } } peers { permanent_uuid: "09233a09a00e42de8081a12cd3be0b01" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 43169 } } peers { permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 } }
I20260619 20:27:32.060889 27214 leader_election.cc:288] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [CANDIDATE]: Term 1 election: Requested vote from peers 2644b5c454124816b2fa5484319aa47a (127.21.151.3:41263), d61bbcd2673e49509cdd5827733ccf1c (127.21.151.2:35061)
I20260619 20:27:32.061022 27146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea76ca960d0476ea84b02e6a354922a" candidate_uuid: "09233a09a00e42de8081a12cd3be0b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2644b5c454124816b2fa5484319aa47a"
I20260619 20:27:32.061036 27015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea76ca960d0476ea84b02e6a354922a" candidate_uuid: "09233a09a00e42de8081a12cd3be0b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d61bbcd2673e49509cdd5827733ccf1c"
I20260619 20:27:32.061101 27015 raft_consensus.cc:3037] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:32.061102 27146 raft_consensus.cc:3037] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:32.061966 27015 raft_consensus.cc:2454] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09233a09a00e42de8081a12cd3be0b01 in term 1.
I20260619 20:27:32.061966 27146 raft_consensus.cc:2454] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09233a09a00e42de8081a12cd3be0b01 in term 1.
I20260619 20:27:32.062141 26818 leader_election.cc:302] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09233a09a00e42de8081a12cd3be0b01, d61bbcd2673e49509cdd5827733ccf1c; no voters:
I20260619 20:27:32.062237 27214 raft_consensus.cc:2781] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:32.062364 27214 raft_consensus.cc:686] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [term 1 LEADER]: Becoming Leader. State: Replica: 09233a09a00e42de8081a12cd3be0b01, State: Running, Role: LEADER
I20260619 20:27:32.062467 27214 consensus_queue.cc:237] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 } } peers { permanent_uuid: "09233a09a00e42de8081a12cd3be0b01" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 43169 } } peers { permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 } }
I20260619 20:27:32.063200 26742 catalog_manager.cc:5495] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09233a09a00e42de8081a12cd3be0b01 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "09233a09a00e42de8081a12cd3be0b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09233a09a00e42de8081a12cd3be0b01" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 43169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 } health_report { overall_health: UNKNOWN } } }
I20260619 20:27:32.077422 27205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.076179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.21.151.1:43169)
0619 20:27:32.076226 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.076228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.076237 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:32.076276 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.076277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.076293 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.076366 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.077166 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.077167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.077168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.077173 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.077193 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.077198 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.077277 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.077302 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.077319 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:27:32.079994 27015 raft_consensus.cc:1261] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c [term 1 FOLLOWER]: Refusing update from remote peer 09233a09a00e42de8081a12cd3be0b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:32.080073 27146 raft_consensus.cc:1261] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a [term 1 FOLLOWER]: Refusing update from remote peer 09233a09a00e42de8081a12cd3be0b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:32.080395 27216 consensus_queue.cc:1035] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2644b5c454124816b2fa5484319aa47a" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 41263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:32.080539 27214 consensus_queue.cc:1035] T 1ea76ca960d0476ea84b02e6a354922a P 09233a09a00e42de8081a12cd3be0b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d61bbcd2673e49509cdd5827733ccf1c" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:32.083371 27219 mvcc.cc:205] Tried to move back new op lower bound from 7298665890116222976 to 7298665890048581632. Current Snapshot: MvccSnapshot[applied={T|T < 7298665890116222976 or (T in {7298665890116222976})}]
I20260619 20:27:32.084115 27221 mvcc.cc:205] Tried to move back new op lower bound from 7298665890116222976 to 7298665890048581632. Current Snapshot: MvccSnapshot[applied={T|T < 7298665890116222976}]
I20260619 20:27:32.085280 26741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:32.085368 26741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:32.086123 26741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3afbc84d699d4de792ad18ef55a698a5: Sending DeleteTablet for 3 replicas of tablet 1ea76ca960d0476ea84b02e6a354922a
I20260619 20:27:32.086547 27126 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea76ca960d0476ea84b02e6a354922a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:32 UTC) from {username='slave'} at 127.0.0.1:41540
I20260619 20:27:32.086771 27229 tablet_replica.cc:330] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: stopping tablet replica
I20260619 20:27:32.086750 26995 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea76ca960d0476ea84b02e6a354922a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:32 UTC) from {username='slave'} at 127.0.0.1:37804
I20260619 20:27:32.086755 26864 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea76ca960d0476ea84b02e6a354922a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:32 UTC) from {username='slave'} at 127.0.0.1:45228
I20260619 20:27:32.086858 27229 raft_consensus.cc:2229] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:32.086953 27229 raft_consensus.cc:2258] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:32.086953 27230 tablet_replica.cc:330] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: stopping tablet replica
I20260619 20:27:32.087047 27230 raft_consensus.cc:2229] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:32.087134 27230 raft_consensus.cc:2258] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:32.087447 27229 ts_tablet_manager.cc:1905] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:32.087530 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 26802
I20260619 20:27:32.089464 27230 ts_tablet_manager.cc:1905] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:32.091280 27230 ts_tablet_manager.cc:1918] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:32.091338 27229 ts_tablet_manager.cc:1918] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:32.091355 27230 log.cc:1196] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/1ea76ca960d0476ea84b02e6a354922a
I20260619 20:27:32.091409 27229 log.cc:1196] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/1ea76ca960d0476ea84b02e6a354922a
I20260619 20:27:32.091650 27230 ts_tablet_manager.cc:1939] T 1ea76ca960d0476ea84b02e6a354922a P d61bbcd2673e49509cdd5827733ccf1c: Deleting consensus metadata
I20260619 20:27:32.091737 27229 ts_tablet_manager.cc:1939] T 1ea76ca960d0476ea84b02e6a354922a P 2644b5c454124816b2fa5484319aa47a: Deleting consensus metadata
I20260619 20:27:32.092036 26727 catalog_manager.cc:4841] TS d61bbcd2673e49509cdd5827733ccf1c (127.21.151.2:35061): tablet 1ea76ca960d0476ea84b02e6a354922a (table test-table [id=2f61c5ddea3d4774bf6189ed6815a180]) successfully deleted
I20260619 20:27:32.092280 26726 catalog_manager.cc:4841] TS 2644b5c454124816b2fa5484319aa47a (127.21.151.3:41263): tablet 1ea76ca960d0476ea84b02e6a354922a (table test-table [id=2f61c5ddea3d4774bf6189ed6815a180]) successfully deleted
W20260619 20:27:32.094615 26729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:43169 (error 108)
I20260619 20:27:32.094841 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 26933
I20260619 20:27:32.095572 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.095423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:43169 (local address 127.0.0.1:45244)
0619 20:27:32.095485 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.095532 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 20:27:32.095700 26729 catalog_manager.cc:4584] TS 09233a09a00e42de8081a12cd3be0b01 (127.21.151.1:43169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ea76ca960d0476ea84b02e6a354922a: Network error: Client connection negotiation failed: client connection to 127.21.151.1:43169: connect: Connection refused (error 111)
I20260619 20:27:32.100617 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 27064
I20260619 20:27:32.104702 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 26711
2026-06-19T20:27:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (598 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 20:27:32.124320 22108 mini_oidc.cc:152] Starting JWKS server
I20260619 20:27:32.125404 22108 webserver.cc:460] Webserver started at https://127.0.0.1:46295/ using document root <none> and password file <none>
I20260619 20:27:32.125517 22108 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 20:27:32.125718 22108 webserver.cc:460] Webserver started at http://127.0.0.1:38409/ using document root <none> and password file <none>
2026-06-19T20:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:32Z Disabled control of system clock
I20260619 20:27:32.133477 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:35687
--webserver_interface=127.21.151.62
--webserver_port=0
--builtin_ntp_servers=127.21.151.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:35687
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46295/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/testchainca.cert with env {}
W20260619 20:27:32.214154 27238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:32.214351 27238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:32.214375 27238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:32.215504 27238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:32.215548 27238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:32.215561 27238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:32.215631 27238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:32.215652 27238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:27:32.215680 27238 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:27:32.215710 27238 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46295/jwks.json
I20260619 20:27:32.217363 27238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:35687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46295/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:32.217623 27238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:32.217864 27238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:32.222788 27243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:32.222790 27246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:32.222779 27244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:32.222844 27238 server_base.cc:1034] running on GCE node
I20260619 20:27:32.223222 27238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:32.223533 27238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:32.224678 27238 hybrid_clock.cc:648] HybridClock initialized: now 1781900852224677 us; error 39 us; skew 500 ppm
I20260619 20:27:32.225814 27238 webserver.cc:460] Webserver started at http://127.21.151.62:43299/ using document root <none> and password file <none>
I20260619 20:27:32.226070 27238 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:32.226115 27238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:32.226265 27238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:32.227363 27238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "1f4036332a6f4ac7b5181a2bc4b37c84"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "b39437ac6b3fa51f66ef11c2f1a66129"
server_key_iv: "ccb323397b8bf0be84e7afea8fad66e8"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.227734 27238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "1f4036332a6f4ac7b5181a2bc4b37c84"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "b39437ac6b3fa51f66ef11c2f1a66129"
server_key_iv: "ccb323397b8bf0be84e7afea8fad66e8"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.228997 27238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:32.229697 27252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:32.229853 27238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:32.229944 27238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "1f4036332a6f4ac7b5181a2bc4b37c84"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "b39437ac6b3fa51f66ef11c2f1a66129"
server_key_iv: "ccb323397b8bf0be84e7afea8fad66e8"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.230034 27238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:32.247891 27238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:32.248183 27238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:32.248328 27238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:32.252024 27238 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:35687
I20260619 20:27:32.252059 27304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:35687 every 8 connection(s)
I20260619 20:27:32.252388 27238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:32.253275 27305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:32.255496 27305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84: Bootstrap starting.
I20260619 20:27:32.256106 27305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:32.256390 27305 log.cc:824] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:32.257045 27305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84: No bootstrap required, opened a new log
I20260619 20:27:32.257622 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 27238
I20260619 20:27:32.257735 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:32.257915 22108 external_mini_cluster.cc:1428] Setting key 99be1d8641158f354cc53be8db8c4b03
I20260619 20:27:32.258955 27305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4036332a6f4ac7b5181a2bc4b37c84" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35687 } }
I20260619 20:27:32.259120 27305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:32.259156 27305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4036332a6f4ac7b5181a2bc4b37c84, State: Initialized, Role: FOLLOWER
I20260619 20:27:32.259289 27305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4036332a6f4ac7b5181a2bc4b37c84" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35687 } }
I20260619 20:27:32.259373 27305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:32.259421 27305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:32.259469 27305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:32.260058 27305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4036332a6f4ac7b5181a2bc4b37c84" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35687 } }
I20260619 20:27:32.260170 27305 leader_election.cc:302] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f4036332a6f4ac7b5181a2bc4b37c84; no voters:
I20260619 20:27:32.260316 27305 leader_election.cc:288] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:32.260354 27310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:32.260491 27310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [term 1 LEADER]: Becoming Leader. State: Replica: 1f4036332a6f4ac7b5181a2bc4b37c84, State: Running, Role: LEADER
I20260619 20:27:32.260576 27305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:32.260572 27310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4036332a6f4ac7b5181a2bc4b37c84" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35687 } }
I20260619 20:27:32.262519 27311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f4036332a6f4ac7b5181a2bc4b37c84. Latest consensus state: current_term: 1 leader_uuid: "1f4036332a6f4ac7b5181a2bc4b37c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4036332a6f4ac7b5181a2bc4b37c84" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35687 } } }
I20260619 20:27:32.262710 27311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:32.262631 27312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f4036332a6f4ac7b5181a2bc4b37c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4036332a6f4ac7b5181a2bc4b37c84" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35687 } } }
I20260619 20:27:32.263000 27312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84 [sys.catalog]: This master's current role is: LEADER
W20260619 20:27:32.263159 27324 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:27:32.263212 27324 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:27:32.263921 27325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:32.264112 27325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:32.264393 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.258386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.21.151.62:35687)
0619 20:27:32.258577 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.258585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.258609 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:27:32.258938 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.258944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.259010 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.259139 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.259422 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.261049 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.261916 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.262549 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.262674 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.263574 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.263577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.263578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.263792 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.263825 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.263836 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.263888 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.264108 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.264187 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260619 20:27:32.265472 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:27:32.265723 27325 catalog_manager.cc:1347] Generated new cluster ID: c8add7909d9c463c9936dcd14e97160a
I20260619 20:27:32.265794 27325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:32.293486 27325 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:32.294059 27325 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:32.298034 27325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f4036332a6f4ac7b5181a2bc4b37c84: Generated new TSK 0
I20260619 20:27:32.298218 27325 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 20:27:32.319228 27335 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 20:27:32.319336 27335 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 20:27:32.319555 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.304066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.21.151.62:35687)
0619 20:27:32.304124 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.304126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.304272 (+ 146us) server_negotiation.cc:407] Connection header received
0619 20:27:32.304375 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.304376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.304386 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.304442 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:27:32.304621 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.305085 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.306197 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.306368 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.306416 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.306639 (+ 223us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:27:32.319342 (+ 12703us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 20:27:32.319506 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 20:27:32.319618 27309 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 20:27:32.320143 27334 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:32.304019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:35687 (local address 127.0.0.1:37654)
0619 20:27:32.304201 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.304213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:32.304286 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:32.304474 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:32.304475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:32.304477 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:27:32.304553 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:32.304556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.305156 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.305157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:32.306120 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:32.306123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.306441 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.306442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:32.306513 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.306536 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:27:32.319780 (+ 13244us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 20:27:32.319893 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.151.62:35687: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.151.62:35687: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46295/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 20:27:32.320745 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 27238
2026-06-19T20:27:32Z chronyd exiting
I20260619 20:27:32.356606 22108 test_util.cc:175] -----------------------------------------------
I20260619 20:27:32.356684 22108 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781900829246020-22108-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 20:27:32.357343 22108 mini_oidc.cc:152] Starting JWKS server
I20260619 20:27:32.357977 22108 webserver.cc:460] Webserver started at https://127.0.0.1:42655/ using document root <none> and password file <none>
I20260619 20:27:32.358016 22108 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 20:27:32.358152 22108 webserver.cc:460] Webserver started at http://127.0.0.1:34449/ using document root <none> and password file <none>
2026-06-19T20:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:32Z Disabled control of system clock
I20260619 20:27:32.365317 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:35163
--webserver_interface=127.21.151.62
--webserver_port=0
--builtin_ntp_servers=127.21.151.20:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:35163
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42655/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/testchainca.cert with env {}
W20260619 20:27:32.446595 27341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:32.446887 27341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:32.446926 27341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:32.448149 27341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:32.448204 27341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:32.448225 27341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:32.448244 27341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:32.448271 27341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:27:32.448294 27341 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:27:32.448323 27341 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42655/jwks.json
I20260619 20:27:32.449816 27341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:35163
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:35163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42655/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:32.450088 27341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:32.450295 27341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:32.455003 27346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:32.455189 27341 server_base.cc:1034] running on GCE node
W20260619 20:27:32.455003 27349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:32.455041 27347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:32.455456 27341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:32.455739 27341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:32.456934 27341 hybrid_clock.cc:648] HybridClock initialized: now 1781900852456917 us; error 28 us; skew 500 ppm
I20260619 20:27:32.458086 27341 webserver.cc:460] Webserver started at http://127.21.151.62:33819/ using document root <none> and password file <none>
I20260619 20:27:32.458303 27341 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:32.458364 27341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:32.458488 27341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:32.459491 27341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "418917267f8d477d95ebb3def43f9783"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "7e2fdda214ee457cc996013de53356f2"
server_key_iv: "73bd98126f475621135f34ce9b2ad40c"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.459872 27341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "418917267f8d477d95ebb3def43f9783"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "7e2fdda214ee457cc996013de53356f2"
server_key_iv: "73bd98126f475621135f34ce9b2ad40c"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.461133 27341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:32.461800 27355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:32.461970 27341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:32.462059 27341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "418917267f8d477d95ebb3def43f9783"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "7e2fdda214ee457cc996013de53356f2"
server_key_iv: "73bd98126f475621135f34ce9b2ad40c"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.462136 27341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:32.469110 27341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:32.469364 27341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:32.469498 27341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:32.473291 27341 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:35163
I20260619 20:27:32.473320 27408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:35163 every 8 connection(s)
I20260619 20:27:32.473623 27341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:32.474195 27409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:32.476718 27409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783: Bootstrap starting.
I20260619 20:27:32.477252 27409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:32.477515 27409 log.cc:824] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:32.478108 27409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783: No bootstrap required, opened a new log
I20260619 20:27:32.479110 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 27341
I20260619 20:27:32.479204 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:32.479374 22108 external_mini_cluster.cc:1428] Setting key 5405f7883ec46f56e3bc2b17cf197cd8
I20260619 20:27:32.480239 27409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418917267f8d477d95ebb3def43f9783" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35163 } }
I20260619 20:27:32.480410 27409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:32.480448 27409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418917267f8d477d95ebb3def43f9783, State: Initialized, Role: FOLLOWER
I20260619 20:27:32.480587 27409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418917267f8d477d95ebb3def43f9783" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35163 } }
I20260619 20:27:32.480688 27409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:32.480721 27409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:32.480769 27409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:32.481449 27409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418917267f8d477d95ebb3def43f9783" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35163 } }
I20260619 20:27:32.481564 27409 leader_election.cc:302] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 418917267f8d477d95ebb3def43f9783; no voters:
I20260619 20:27:32.481702 27409 leader_election.cc:288] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:32.481727 27414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:32.481884 27414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [term 1 LEADER]: Becoming Leader. State: Replica: 418917267f8d477d95ebb3def43f9783, State: Running, Role: LEADER
I20260619 20:27:32.481913 27409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:32.481986 27414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418917267f8d477d95ebb3def43f9783" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35163 } }
I20260619 20:27:32.482299 27415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "418917267f8d477d95ebb3def43f9783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418917267f8d477d95ebb3def43f9783" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35163 } } }
I20260619 20:27:32.482390 27415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:32.482290 27414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 418917267f8d477d95ebb3def43f9783. Latest consensus state: current_term: 1 leader_uuid: "418917267f8d477d95ebb3def43f9783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418917267f8d477d95ebb3def43f9783" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 35163 } } }
I20260619 20:27:32.482571 27414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:32.482686 27422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:32.483183 27422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:32.485399 27422 catalog_manager.cc:1347] Generated new cluster ID: 8ca161b4e96240619b6b050347b5bf13
I20260619 20:27:32.485458 27422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:32.485914 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.479897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.21.151.62:35163)
0619 20:27:32.480083 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.480090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.480113 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:27:32.480202 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.480207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.480262 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.480387 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.480655 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.482451 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.483665 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.484253 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.484361 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.485177 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.485181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.485181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.485391 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.485436 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.485452 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.485528 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.485632 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.485706 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260619 20:27:32.486735 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:27:32.502512 27422 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:32.502974 27422 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:32.508900 27422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 418917267f8d477d95ebb3def43f9783: Generated new TSK 0
I20260619 20:27:32.509095 27422 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 20:27:32.528995 27439 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 20:27:32.529075 27439 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 20:27:32.529410 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.515210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.21.151.62:35163)
0619 20:27:32.515250 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.515252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.515303 (+ 51us) server_negotiation.cc:407] Connection header received
0619 20:27:32.515391 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.515392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.515402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.515466 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:27:32.515594 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.516122 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.517309 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.517617 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.517655 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.517883 (+ 228us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:27:32.529196 (+ 11313us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 20:27:32.529338 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 20:27:32.529434 27438 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:32.515090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:35163 (local address 127.0.0.1:45636)
0619 20:27:32.515245 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.515257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:32.515323 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:32.515473 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:32.515475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:32.515476 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:27:32.515532 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:32.515534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.516222 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.516223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:32.517204 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:32.517207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.517716 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.517717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:32.517778 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.517783 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:27:32.529291 (+ 11508us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 20:27:32.529370 (+ 79us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.151.62:35163: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
W20260619 20:27:32.529470 27413 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 20:27:32.529702 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 27341
2026-06-19T20:27:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 20:27:32.567268 22108 mini_oidc.cc:152] Starting JWKS server
I20260619 20:27:32.567960 22108 webserver.cc:460] Webserver started at https://127.0.0.1:32937/ using document root <none> and password file <none>
I20260619 20:27:32.568009 22108 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 20:27:32.568114 22108 webserver.cc:460] Webserver started at http://127.0.0.1:43025/ using document root <none> and password file <none>
2026-06-19T20:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:32Z Disabled control of system clock
I20260619 20:27:32.575714 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44173
--webserver_interface=127.21.151.62
--webserver_port=0
--builtin_ntp_servers=127.21.151.20:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:44173
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32937/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 20:27:32.653192 27445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:32.653386 27445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:32.653407 27445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:32.654481 27445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:32.654520 27445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:32.654533 27445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:32.654546 27445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:32.654558 27445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:27:32.654570 27445 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:27:32.654588 27445 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32937/jwks.json
I20260619 20:27:32.655860 27445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:44173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32937/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:32.656038 27445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:32.656255 27445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:32.660779 27450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:32.660831 27451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:32.660779 27453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:32.660952 27445 server_base.cc:1034] running on GCE node
I20260619 20:27:32.661212 27445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:32.661444 27445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:32.662598 27445 hybrid_clock.cc:648] HybridClock initialized: now 1781900852662589 us; error 28 us; skew 500 ppm
I20260619 20:27:32.663826 27445 webserver.cc:460] Webserver started at http://127.21.151.62:33545/ using document root <none> and password file <none>
I20260619 20:27:32.664047 27445 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:32.664126 27445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:32.664266 27445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:32.665292 27445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "cc2e209a1bf2480699e52da2d5698e91"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "9373bb0053785a9a84b2c4083afd0ea0"
server_key_iv: "9c6c819f2388c37d20aa4e080accb5d6"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.665642 27445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "cc2e209a1bf2480699e52da2d5698e91"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "9373bb0053785a9a84b2c4083afd0ea0"
server_key_iv: "9c6c819f2388c37d20aa4e080accb5d6"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.666929 27445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:27:32.667604 27459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:32.667797 27445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:32.667887 27445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "cc2e209a1bf2480699e52da2d5698e91"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "9373bb0053785a9a84b2c4083afd0ea0"
server_key_iv: "9c6c819f2388c37d20aa4e080accb5d6"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.667968 27445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:32.681214 27445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:32.681461 27445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:32.681602 27445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:32.685348 27445 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:44173
I20260619 20:27:32.685377 27512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:44173 every 8 connection(s)
I20260619 20:27:32.685715 27445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:32.686340 27513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:32.689074 27513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91: Bootstrap starting.
I20260619 20:27:32.689640 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 27445
I20260619 20:27:32.689720 27513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:32.689749 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:32.689936 22108 external_mini_cluster.cc:1428] Setting key b959912a795270b0ae98ee2210d7248a
I20260619 20:27:32.690047 27513 log.cc:824] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:32.690842 27513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91: No bootstrap required, opened a new log
I20260619 20:27:32.692940 27513 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2e209a1bf2480699e52da2d5698e91" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44173 } }
I20260619 20:27:32.693097 27513 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:32.693140 27513 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2e209a1bf2480699e52da2d5698e91, State: Initialized, Role: FOLLOWER
I20260619 20:27:32.693272 27513 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2e209a1bf2480699e52da2d5698e91" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44173 } }
I20260619 20:27:32.693358 27513 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:32.693400 27513 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:32.693447 27513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:32.693930 27513 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2e209a1bf2480699e52da2d5698e91" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44173 } }
I20260619 20:27:32.694046 27513 leader_election.cc:302] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc2e209a1bf2480699e52da2d5698e91; no voters:
I20260619 20:27:32.694199 27513 leader_election.cc:288] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:32.694231 27518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:32.694358 27518 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [term 1 LEADER]: Becoming Leader. State: Replica: cc2e209a1bf2480699e52da2d5698e91, State: Running, Role: LEADER
I20260619 20:27:32.694435 27513 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:32.694446 27518 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2e209a1bf2480699e52da2d5698e91" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44173 } }
I20260619 20:27:32.694794 27518 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc2e209a1bf2480699e52da2d5698e91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2e209a1bf2480699e52da2d5698e91" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44173 } } }
I20260619 20:27:32.694862 27518 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:32.695012 27518 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc2e209a1bf2480699e52da2d5698e91. Latest consensus state: current_term: 1 leader_uuid: "cc2e209a1bf2480699e52da2d5698e91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2e209a1bf2480699e52da2d5698e91" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44173 } } }
I20260619 20:27:32.695073 27518 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:32.695349 27524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:32.695541 27517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.690424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.21.151.62:44173)
0619 20:27:32.690620 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.690627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.690654 (+ 27us) server_negotiation.cc:407] Connection header received
0619 20:27:32.691140 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.691147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.691204 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.691312 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.691555 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.692470 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.693295 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.693777 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.693900 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.694605 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.694609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.694609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.694828 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.694869 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.694883 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.694973 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.695216 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.695302 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260619 20:27:32.695855 27524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:32.696691 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:27:32.697300 27524 catalog_manager.cc:1347] Generated new cluster ID: f77036e024324fd385ffce6b78d1f8f4
I20260619 20:27:32.697350 27524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:32.705197 27524 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:32.705861 27524 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:32.713292 27524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc2e209a1bf2480699e52da2d5698e91: Generated new TSK 0
I20260619 20:27:32.713455 27524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:27:32.730427 27517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.726491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.21.151.62:44173)
0619 20:27:32.726544 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.726547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.726612 (+ 65us) server_negotiation.cc:407] Connection header received
0619 20:27:32.726677 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.726686 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.726705 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.726807 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:27:32.726897 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.727800 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.728571 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.728919 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.729007 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.729128 (+ 121us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:27:32.730190 (+ 1062us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32937/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 20:27:32.730346 (+ 156us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49040: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32937/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 20:27:32.730499 27517 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49040: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32937/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 20:27:32.732326 27542 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:32.726401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44173 (local address 127.0.0.1:49040)
0619 20:27:32.726560 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.726569 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:32.726620 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:32.726803 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:32.726803 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:32.726805 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:27:32.726850 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:32.726852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.727896 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.727896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:32.728488 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:32.728489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.729004 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.729005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:32.729043 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.729047 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:27:32.730292 (+ 1245us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32937/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 20:27:32.732227 (+ 1935us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.151.62:44173: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32937/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1765,"thread_start_us":104,"threads_started":1}
I20260619 20:27:32.732467 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 27445
E20260619 20:27:32.741626 27543 webserver.cc:533] Webserver: sslize failed:
E20260619 20:27:32.741690 27543 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T20:27:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (202 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T20:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:32Z Disabled control of system clock
I20260619 20:27:32.777235 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:33891
--webserver_interface=127.21.151.62
--webserver_port=0
--builtin_ntp_servers=127.21.151.20:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:33891
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/testchainca.cert with env {}
W20260619 20:27:32.854715 27547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:32.854988 27547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:32.855011 27547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:32.856287 27547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:32.856329 27547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:32.856343 27547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:32.856355 27547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:32.856371 27547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 20:27:32.856388 27547 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 20:27:32.856405 27547 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 20:27:32.857831 27547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:33891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:33891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:32.858016 27547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:32.858162 27547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:32.862646 27552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:32.862660 27553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:32.862634 27555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:32.862852 27547 server_base.cc:1034] running on GCE node
I20260619 20:27:32.863054 27547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:32.863317 27547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:32.864466 27547 hybrid_clock.cc:648] HybridClock initialized: now 1781900852864460 us; error 35 us; skew 500 ppm
I20260619 20:27:32.865551 27547 webserver.cc:460] Webserver started at http://127.21.151.62:40147/ using document root <none> and password file <none>
I20260619 20:27:32.865778 27547 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:32.865844 27547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:32.865993 27547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:32.867019 27547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "c7f991380be5428a9386be4f4c7330eb"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "69843f9ed7f58aa92bad06b3772e35c8"
server_key_iv: "f9102f0f83807c977747af8f1569a479"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.867357 27547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "c7f991380be5428a9386be4f4c7330eb"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "69843f9ed7f58aa92bad06b3772e35c8"
server_key_iv: "f9102f0f83807c977747af8f1569a479"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.868667 27547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:32.869398 27561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:32.869578 27547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:32.869671 27547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "c7f991380be5428a9386be4f4c7330eb"
format_stamp: "Formatted at 2026-06-19 20:27:32 on dist-test-slave-f9rk"
server_key: "69843f9ed7f58aa92bad06b3772e35c8"
server_key_iv: "f9102f0f83807c977747af8f1569a479"
server_key_version: "encryptionkey@0"
I20260619 20:27:32.869751 27547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:32.886845 27547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:32.887146 27547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:32.887288 27547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:32.891283 27547 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:33891
I20260619 20:27:32.891330 27615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:33891 every 8 connection(s)
I20260619 20:27:32.891721 27547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:32.892702 27616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:32.894907 27616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb: Bootstrap starting.
I20260619 20:27:32.895500 27616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:32.895809 27616 log.cc:824] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:32.896469 27616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb: No bootstrap required, opened a new log
I20260619 20:27:32.898021 27616 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f991380be5428a9386be4f4c7330eb" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33891 } }
I20260619 20:27:32.898154 27616 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:32.898196 27616 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f991380be5428a9386be4f4c7330eb, State: Initialized, Role: FOLLOWER
I20260619 20:27:32.898340 27616 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f991380be5428a9386be4f4c7330eb" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33891 } }
I20260619 20:27:32.898416 27616 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:32.898463 27616 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:32.898512 27616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:32.899029 27616 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f991380be5428a9386be4f4c7330eb" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33891 } }
I20260619 20:27:32.899130 27616 leader_election.cc:302] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7f991380be5428a9386be4f4c7330eb; no voters:
I20260619 20:27:32.899299 27616 leader_election.cc:288] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:32.899385 27619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:32.899523 27619 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [term 1 LEADER]: Becoming Leader. State: Replica: c7f991380be5428a9386be4f4c7330eb, State: Running, Role: LEADER
I20260619 20:27:32.899541 27616 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:32.899693 27619 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f991380be5428a9386be4f4c7330eb" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33891 } }
I20260619 20:27:32.900051 27620 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7f991380be5428a9386be4f4c7330eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f991380be5428a9386be4f4c7330eb" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33891 } } }
I20260619 20:27:32.900128 27620 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:32.900233 27621 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7f991380be5428a9386be4f4c7330eb. Latest consensus state: current_term: 1 leader_uuid: "c7f991380be5428a9386be4f4c7330eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f991380be5428a9386be4f4c7330eb" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33891 } } }
I20260619 20:27:32.900362 27621 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:32.900485 27628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:32.900990 27628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:32.901270 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 27547
I20260619 20:27:32.901368 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:32.901538 22108 external_mini_cluster.cc:1428] Setting key 43ae15b4fddfa08301872c995d041fe2
I20260619 20:27:32.902506 27628 catalog_manager.cc:1347] Generated new cluster ID: 66d208322e1f40bcb217af3544581439
I20260619 20:27:32.902559 27628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:32.907284 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.901955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.21.151.62:33891)
0619 20:27:32.902151 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.902160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.902197 (+ 37us) server_negotiation.cc:407] Connection header received
0619 20:27:32.902309 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.902316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.902383 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.902530 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:32.902862 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.904201 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.905261 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.905838 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.905942 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.906618 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:32.906621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:32.906622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:27:32.906815 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:32.906852 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:27:32.906864 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:32.906920 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:32.907022 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:27:32.907086 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260619 20:27:32.908290 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:27:32.910346 27628 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:32.910796 27628 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:32.915297 27628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7f991380be5428a9386be4f4c7330eb: Generated new TSK 0
I20260619 20:27:32.915445 27628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:27:33.354995 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:32.919838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.21.151.62:33891)
0619 20:27:32.919904 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:32.919907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:32.919981 (+ 74us) server_negotiation.cc:407] Connection header received
0619 20:27:32.920052 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:32.920052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:32.920064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:32.920132 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0619 20:27:32.920248 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.920713 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.921483 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.921679 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.921746 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.921926 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 20:27:33.354697 (+432771us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 20:27:33.354906 (+ 209us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32880: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 20:27:33.355087 27645 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:32.919796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33891 (local address 127.0.0.1:32880)
0619 20:27:32.919934 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:32.919943 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:32.919997 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:32.920132 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:32.920133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:32.920134 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 20:27:32.920190 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:32.920192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.920804 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.920805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:32.921394 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:32.921396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:32.921746 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:32.921747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:32.921804 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:32.921808 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 20:27:33.354856 (+433048us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 20:27:33.354972 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.151.62:33891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
W20260619 20:27:33.355103 27638 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32880: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260619 20:27:33.355407 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 27547
2026-06-19T20:27:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (602 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:33 dist-test-slave-f9rk krb5kdc[27657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:33 dist-test-slave-f9rk krb5kdc[27657](info): set up 2 sockets
Jun 19 20:27:33 dist-test-slave-f9rk krb5kdc[27657](info): commencing operation
krb5kdc: starting...
W20260619 20:27:34.557421 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:34 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:34.669900 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:46759
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:46759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:34.747165 27670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:34.747365 27670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:34.747397 27670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:34.748577 27670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:34.748639 27670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:34.748662 27670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:34.748683 27670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:34.748711 27670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:34.750100 27670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:46759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:46759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:34.750353 27670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:34.750562 27670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:34.755267 27675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:34.755277 27678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:34.755336 27670 server_base.cc:1034] running on GCE node
W20260619 20:27:34.755277 27676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:34.755757 27670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:34.756014 27670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:34.757167 27670 hybrid_clock.cc:648] HybridClock initialized: now 1781900854757156 us; error 28 us; skew 500 ppm
Jun 19 20:27:34 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900854, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:34.759543 27670 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:34.759968 27670 webserver.cc:460] Webserver started at http://127.21.151.62:37801/ using document root <none> and password file <none>
I20260619 20:27:34.760212 27670 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:34.760250 27670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:34.760334 27670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:34.761320 27670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "627be115377c4cc3a17af3934751051b"
format_stamp: "Formatted at 2026-06-19 20:27:34 on dist-test-slave-f9rk"
server_key: "26b5240f5b4b39e1b74dd64a78a721b5"
server_key_iv: "6fe9bcf55cec7a75a902147535663e72"
server_key_version: "encryptionkey@0"
I20260619 20:27:34.761617 27670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "627be115377c4cc3a17af3934751051b"
format_stamp: "Formatted at 2026-06-19 20:27:34 on dist-test-slave-f9rk"
server_key: "26b5240f5b4b39e1b74dd64a78a721b5"
server_key_iv: "6fe9bcf55cec7a75a902147535663e72"
server_key_version: "encryptionkey@0"
I20260619 20:27:34.762799 27670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:34.763438 27685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:34.763612 27670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:34.763747 27670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "627be115377c4cc3a17af3934751051b"
format_stamp: "Formatted at 2026-06-19 20:27:34 on dist-test-slave-f9rk"
server_key: "26b5240f5b4b39e1b74dd64a78a721b5"
server_key_iv: "6fe9bcf55cec7a75a902147535663e72"
server_key_version: "encryptionkey@0"
I20260619 20:27:34.763832 27670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:34.771123 27670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:34.772841 27670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:34.773000 27670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:34.776778 27670 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:46759
I20260619 20:27:34.776806 27738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:46759 every 8 connection(s)
I20260619 20:27:34.777119 27670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:34.778059 27739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:34.780284 27739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b: Bootstrap starting.
I20260619 20:27:34.780850 27739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:34.781131 27739 log.cc:824] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:34.781762 27739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b: No bootstrap required, opened a new log
I20260619 20:27:34.783293 27739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627be115377c4cc3a17af3934751051b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46759 } }
I20260619 20:27:34.783420 27739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:34.783473 27739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627be115377c4cc3a17af3934751051b, State: Initialized, Role: FOLLOWER
I20260619 20:27:34.783665 27739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627be115377c4cc3a17af3934751051b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46759 } }
I20260619 20:27:34.783726 27739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:34.783746 27739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:34.783808 27739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:34.784116 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 27670
I20260619 20:27:34.784194 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:34.784368 22108 external_mini_cluster.cc:1428] Setting key 0c9f0e25716113cb9d67fc60528d0b9f
I20260619 20:27:34.784381 27739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627be115377c4cc3a17af3934751051b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46759 } }
I20260619 20:27:34.784499 27739 leader_election.cc:302] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 627be115377c4cc3a17af3934751051b; no voters:
I20260619 20:27:34.784669 27739 leader_election.cc:288] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:34.784703 27742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:34.784914 27739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:34.785035 27742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [term 1 LEADER]: Becoming Leader. State: Replica: 627be115377c4cc3a17af3934751051b, State: Running, Role: LEADER
I20260619 20:27:34.785128 27742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627be115377c4cc3a17af3934751051b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46759 } }
I20260619 20:27:34.785444 27742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 627be115377c4cc3a17af3934751051b. Latest consensus state: current_term: 1 leader_uuid: "627be115377c4cc3a17af3934751051b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627be115377c4cc3a17af3934751051b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46759 } } }
I20260619 20:27:34.785517 27742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:34.785763 27753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:34.786244 27753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:34.785454 27743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "627be115377c4cc3a17af3934751051b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "627be115377c4cc3a17af3934751051b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46759 } } }
I20260619 20:27:34.786608 27743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:34.787781 27753 catalog_manager.cc:1347] Generated new cluster ID: b5fec2a76eeb410a812f16c2e4b8a799
I20260619 20:27:34.787832 27753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:27:34 dist-test-slave-f9rk krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:34.793282 27753 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:34.793917 27753 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:34.794705 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:34.784801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.21.151.62:46759)
0619 20:27:34.785125 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:34.785134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:34.785171 (+ 37us) server_negotiation.cc:407] Connection header received
0619 20:27:34.785277 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:34.785284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:34.785336 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:34.785487 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:34.786326 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:34.787697 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:34.788478 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:34.789053 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:34.789153 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:34.791270 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:34.791273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:34.791274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:34.791283 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:34.793056 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:34.793478 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:34.793481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:34.793482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:34.793619 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:34.793822 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:34.793823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:34.793824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:34.794106 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:34.794247 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:34.794377 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 20:27:34.794518 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
I20260619 20:27:34.800007 27753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 627be115377c4cc3a17af3934751051b: Generated new TSK 0
I20260619 20:27:34.800168 27753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:27:34.832252 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:46759
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:34.912686 27765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:34.912873 27765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:34.912905 27765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:34.913993 27765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:34.914029 27765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:34.914076 27765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:34.915491 27765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:34.915731 27765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:34.915956 27765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:34.920717 27770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:34.920717 27773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:34.920733 27765 server_base.cc:1034] running on GCE node
W20260619 20:27:34.920733 27771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:34.921149 27765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:34.921379 27765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:34.922520 27765 hybrid_clock.cc:648] HybridClock initialized: now 1781900854922509 us; error 27 us; skew 500 ppm
Jun 19 20:27:34 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900854, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:34.925031 27765 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:27:34.925442 27765 webserver.cc:460] Webserver started at http://127.21.151.1:33659/ using document root <none> and password file <none>
I20260619 20:27:34.925652 27765 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:34.925694 27765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:34.925825 27765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:34.926800 27765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "9c927408877549f99976a6655af5e052"
format_stamp: "Formatted at 2026-06-19 20:27:34 on dist-test-slave-f9rk"
server_key: "7d11c9d099980955b91eed8066d6037a"
server_key_iv: "f653fff3a6852bd24fc644db0a5d4d4b"
server_key_version: "encryptionkey@0"
I20260619 20:27:34.927127 27765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "9c927408877549f99976a6655af5e052"
format_stamp: "Formatted at 2026-06-19 20:27:34 on dist-test-slave-f9rk"
server_key: "7d11c9d099980955b91eed8066d6037a"
server_key_iv: "f653fff3a6852bd24fc644db0a5d4d4b"
server_key_version: "encryptionkey@0"
I20260619 20:27:34.928402 27765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:34.929004 27780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:34.929154 27765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:34.929229 27765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "9c927408877549f99976a6655af5e052"
format_stamp: "Formatted at 2026-06-19 20:27:34 on dist-test-slave-f9rk"
server_key: "7d11c9d099980955b91eed8066d6037a"
server_key_iv: "f653fff3a6852bd24fc644db0a5d4d4b"
server_key_version: "encryptionkey@0"
I20260619 20:27:34.929296 27765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:34.953487 27765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:34.955809 27765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:34.955963 27765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:34.956192 27765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:34.956499 27765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:34.956547 27765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:34.956580 27765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:34.956593 27765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:34.962149 27765 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:42441
I20260619 20:27:34.962196 27894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:42441 every 8 connection(s)
I20260619 20:27:34.962507 27765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:27:34.966790 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 27765
I20260619 20:27:34.966904 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:34.967088 22108 external_mini_cluster.cc:1428] Setting key 573be3fab3b2237f9334c7aa4cfc2950
Jun 19 20:27:34 dist-test-slave-f9rk krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900854, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:34.972363 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:34.963198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:44055 (local address 127.21.151.62:46759)
0619 20:27:34.963243 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:34.963245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:34.963423 (+ 178us) server_negotiation.cc:407] Connection header received
0619 20:27:34.964430 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:34.964432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:34.964443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:34.964501 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:34.965109 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:34.965618 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:34.966620 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:34.966838 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:34.966912 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:34.969492 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:34.969494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:34.969494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:34.969501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:34.970868 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:34.971380 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:34.971380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:34.971381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:34.971430 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:34.971732 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:34.971733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:34.971733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:34.971836 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:34.971894 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:34.972237 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 20:27:34.972304 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:27:34.972672 27898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:34.963159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:46759 (local address 127.21.151.1:44055)
0619 20:27:34.963300 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:34.963342 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:34.964310 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:34.964581 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:34.964585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:34.964828 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:34.965016 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:34.965021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:34.965725 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:34.965726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:34.966536 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:34.966538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:34.966917 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:34.966918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:34.967042 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:34.967561 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:34.967570 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:34.969397 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:34.970966 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:34.970973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:34.970975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:34.971298 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:34.971495 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:34.971496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:34.971496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:34.971665 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:34.971901 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:34.971907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:34.972090 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 20:27:34.972209 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 20:27:34.972386 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260619 20:27:34.973182 27895 heartbeater.cc:347] Connected to a master server at 127.21.151.62:46759
I20260619 20:27:34.973295 27895 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:34.973507 27895 heartbeater.cc:511] Master 127.21.151.62:46759 requested a full tablet report, sending...
I20260619 20:27:34.973992 27702 ts_manager.cc:194] Registered new tserver with Master: 9c927408877549f99976a6655af5e052 (127.21.151.1:42441)
I20260619 20:27:34.974598 27702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:44055
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:27:34.991822 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:46759
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:35.073019 27901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:35.073328 27901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:35.073366 27901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:35.075246 27901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:35.075304 27901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:35.075345 27901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:35.077572 27901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:35.077836 27901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:35.078092 27901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:35.084079 27906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:35.084110 27907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:35.084086 27909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:35.084371 27901 server_base.cc:1034] running on GCE node
I20260619 20:27:35.084555 27901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:35.084776 27901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:35.085922 27901 hybrid_clock.cc:648] HybridClock initialized: now 1781900855085904 us; error 46 us; skew 500 ppm
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:35.088524 27901 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:27:35.088873 27901 webserver.cc:460] Webserver started at http://127.21.151.2:37975/ using document root <none> and password file <none>
I20260619 20:27:35.089080 27901 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:35.089118 27901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:35.089197 27901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:35.090090 27901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "44e0ab58d25349c0b5a9b222df7519ad"
format_stamp: "Formatted at 2026-06-19 20:27:35 on dist-test-slave-f9rk"
server_key: "970fe5478983861b5b0977d3f257671d"
server_key_iv: "69d35af77010a0901c0083268619da81"
server_key_version: "encryptionkey@0"
I20260619 20:27:35.090351 27901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "44e0ab58d25349c0b5a9b222df7519ad"
format_stamp: "Formatted at 2026-06-19 20:27:35 on dist-test-slave-f9rk"
server_key: "970fe5478983861b5b0977d3f257671d"
server_key_iv: "69d35af77010a0901c0083268619da81"
server_key_version: "encryptionkey@0"
I20260619 20:27:35.091503 27901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:27:35.092221 27916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:35.092378 27901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:35.092455 27901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "44e0ab58d25349c0b5a9b222df7519ad"
format_stamp: "Formatted at 2026-06-19 20:27:35 on dist-test-slave-f9rk"
server_key: "970fe5478983861b5b0977d3f257671d"
server_key_iv: "69d35af77010a0901c0083268619da81"
server_key_version: "encryptionkey@0"
I20260619 20:27:35.092519 27901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:35.104333 27901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:35.106038 27901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:35.106178 27901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:35.106408 27901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:35.106693 27901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:35.106726 27901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:35.106763 27901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:35.106791 27901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:35.112270 27901 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:38883
I20260619 20:27:35.112334 28029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:38883 every 8 connection(s)
I20260619 20:27:35.112623 27901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:27:35.116150 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 27901
I20260619 20:27:35.116247 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:35.116422 22108 external_mini_cluster.cc:1428] Setting key bd25cf6da3a9ac3171235df9d87d4d37
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:35.121999 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:35.113321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:44721 (local address 127.21.151.62:46759)
0619 20:27:35.113365 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:35.113367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:35.113575 (+ 208us) server_negotiation.cc:407] Connection header received
0619 20:27:35.114517 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:35.114519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:35.114531 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:35.114584 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:35.115188 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.115729 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.116818 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.117034 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.117115 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:35.119202 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:35.119203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:35.119203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:35.119211 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:35.120584 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:35.121074 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:35.121075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:35.121075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:35.121125 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:35.121395 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:35.121395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:35.121395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:35.121490 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:35.121538 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:35.121865 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 20:27:35.121930 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:35.122294 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:35.113294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:46759 (local address 127.21.151.2:44721)
0619 20:27:35.113464 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:35.113498 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:35.114405 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:35.114650 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:35.114654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:35.114920 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:35.115098 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:35.115102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.115822 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.115823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:35.116728 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:35.116731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.117134 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.117135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:35.117252 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:35.117726 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:35.117734 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:35.119107 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:35.120685 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:35.120692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:35.120694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:35.121001 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:35.121187 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:35.121187 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:35.121188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:35.121327 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:35.121551 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:35.121557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:35.121727 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 20:27:35.121855 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 20:27:35.122026 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260619 20:27:35.122799 28030 heartbeater.cc:347] Connected to a master server at 127.21.151.62:46759
I20260619 20:27:35.122897 28030 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:35.123073 28030 heartbeater.cc:511] Master 127.21.151.62:46759 requested a full tablet report, sending...
I20260619 20:27:35.123417 27702 ts_manager.cc:194] Registered new tserver with Master: 44e0ab58d25349c0b5a9b222df7519ad (127.21.151.2:38883)
I20260619 20:27:35.123833 27702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:44721
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:27:35.141749 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:46759
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:35.220628 28035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:35.220830 28035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:35.220852 28035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:35.221918 28035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:35.221951 28035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:35.221975 28035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:35.223302 28035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:46759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:35.223471 28035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:35.223647 28035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:35.228246 28044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:35.228298 28042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:35.228324 28041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:35.228566 28035 server_base.cc:1034] running on GCE node
I20260619 20:27:35.228783 28035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:35.229032 28035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:35.230176 28035 hybrid_clock.cc:648] HybridClock initialized: now 1781900855230166 us; error 31 us; skew 500 ppm
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:35.232820 28035 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:27:35.233233 28035 webserver.cc:460] Webserver started at http://127.21.151.3:44767/ using document root <none> and password file <none>
I20260619 20:27:35.233428 28035 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:35.233466 28035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:35.233551 28035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:35.234489 28035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "445a828730b6445a8ddf1d30aadac476"
format_stamp: "Formatted at 2026-06-19 20:27:35 on dist-test-slave-f9rk"
server_key: "9c21e9b59a17e8d3ede41bbb4ee1b2b4"
server_key_iv: "207cae248364a74074df32ddeb9e42b4"
server_key_version: "encryptionkey@0"
I20260619 20:27:35.234772 28035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "445a828730b6445a8ddf1d30aadac476"
format_stamp: "Formatted at 2026-06-19 20:27:35 on dist-test-slave-f9rk"
server_key: "9c21e9b59a17e8d3ede41bbb4ee1b2b4"
server_key_iv: "207cae248364a74074df32ddeb9e42b4"
server_key_version: "encryptionkey@0"
I20260619 20:27:35.236126 28035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:27:35.236735 28051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:35.236917 28035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:35.237018 28035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "445a828730b6445a8ddf1d30aadac476"
format_stamp: "Formatted at 2026-06-19 20:27:35 on dist-test-slave-f9rk"
server_key: "9c21e9b59a17e8d3ede41bbb4ee1b2b4"
server_key_iv: "207cae248364a74074df32ddeb9e42b4"
server_key_version: "encryptionkey@0"
I20260619 20:27:35.237090 28035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:35.258107 28035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:35.260850 28035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:35.260972 28035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:35.261166 28035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:35.261445 28035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:35.261482 28035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:35.261505 28035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:35.261523 28035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:35.267352 28035 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:45157
I20260619 20:27:35.267506 28166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:45157 every 8 connection(s)
I20260619 20:27:35.267763 28035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:35.276144 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 28035
I20260619 20:27:35.276252 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:35.276482 22108 external_mini_cluster.cc:1428] Setting key b60bc39fb03dc2f9c7ce319164cb989e
I20260619 20:27:35.276847 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:35.268453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:33737 (local address 127.21.151.62:46759)
0619 20:27:35.268529 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:35.268531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:35.268738 (+ 207us) server_negotiation.cc:407] Connection header received
0619 20:27:35.269671 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:35.269672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:35.269682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:35.269724 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:35.270281 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.270765 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.271777 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.271967 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.272020 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:35.274140 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:35.274141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:35.274142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:35.274149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:35.275338 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:35.275798 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:35.275799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:35.275800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:35.275860 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:35.276112 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:35.276113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:35.276113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:35.276229 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:35.276311 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:35.276653 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 20:27:35.276744 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:27:35.277029 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:35.268441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:46759 (local address 127.21.151.3:33737)
0619 20:27:35.268618 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:35.268660 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:35.269566 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:35.269778 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:35.269782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:35.270030 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:35.270216 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:35.270220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.270836 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.270837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:35.271709 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:35.271711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.272033 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.272034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:35.272129 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:35.272545 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:35.272553 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:35.274015 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:35.275432 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:35.275438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:35.275440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:35.275726 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:35.275925 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:35.275926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:35.275927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:35.276041 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:35.276314 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:35.276320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:35.276493 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 20:27:35.276612 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 20:27:35.276776 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":112,"threads_started":1}
I20260619 20:27:35.277531 28167 heartbeater.cc:347] Connected to a master server at 127.21.151.62:46759
I20260619 20:27:35.277626 28167 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:35.277767 28167 heartbeater.cc:511] Master 127.21.151.62:46759 requested a full tablet report, sending...
I20260619 20:27:35.278100 27701 ts_manager.cc:194] Registered new tserver with Master: 445a828730b6445a8ddf1d30aadac476 (127.21.151.3:45157)
I20260619 20:27:35.278518 27701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:33737
I20260619 20:27:35.288415 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/test-admin.keytab.
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:35.303516 22108 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 20:27:35.305184 28174 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:35.304362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.21.151.1:42441)
0619 20:27:35.304538 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:35.304543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:35.304551 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:35.304574 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:35.304575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:35.304588 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:35.304667 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:35.305100 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42240: BlockingRecv error: recv got EOF from 127.0.0.1:42240 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
W20260619 20:27:35.305259 22108 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5cc' of type FILE
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:35.306241 22108 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:35.307096 22108 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.1@KRBTEST.COM
I20260619 20:27:35.314267 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:35.307579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.21.151.1:42441)
0619 20:27:35.307648 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:35.307650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:35.307659 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:35.307736 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:35.307737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:35.307746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:35.307802 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:35.308185 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.308901 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.309633 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.309940 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.310037 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:35.312020 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:35.312027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:35.312028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:35.312034 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:35.313331 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:35.313624 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:35.313627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:35.313627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:35.313677 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:35.313812 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:35.313812 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:35.313812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:35.313916 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:35.314003 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:35.314064 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 20:27:35.314122 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 20:27:35.315836 28174 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:35.315316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.21.151.1:42441)
0619 20:27:35.315370 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:35.315372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:35.315445 (+ 73us) server_negotiation.cc:407] Connection header received
0619 20:27:35.315487 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:35.315487 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:35.315496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:35.315532 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:35.315787 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42260: BlockingRecv error: recv got EOF from 127.0.0.1:42260 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 20:27:35.315907 22108 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5cc' of type FILE
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:35.316772 22108 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:35.317651 22108 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.1@KRBTEST.COM
I20260619 20:27:35.323462 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:35.318053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.21.151.1:42441)
0619 20:27:35.318095 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:35.318097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:35.318120 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:27:35.318166 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:35.318166 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:35.318173 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:35.318216 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:35.318500 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.318943 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.319567 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.319815 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.319867 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:35.321607 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:35.321608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:35.321608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:35.321613 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:35.322710 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:35.322985 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:35.322986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:35.322986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:35.323032 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:35.323168 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:35.323168 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:35.323168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:35.323238 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:35.323297 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:35.323360 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 20:27:35.323411 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260619 20:27:35.324853 28174 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:35.324485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42266 (local address 127.21.151.1:42441)
0619 20:27:35.324521 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:35.324523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:35.324544 (+ 21us) server_negotiation.cc:407] Connection header received
0619 20:27:35.324585 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:35.324586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:35.324593 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:35.324630 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:35.324825 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42266: BlockingRecv error: recv got EOF from 127.0.0.1:42266 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 20:27:35.324957 22108 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781900829246020-22108-0/krb5kdc/krb5cc' of type FILE
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:35.325843 22108 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:35.326628 22108 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.1@KRBTEST.COM
I20260619 20:27:35.332854 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:35.327065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42276 (local address 127.21.151.1:42441)
0619 20:27:35.327135 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:35.327137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:35.327182 (+ 45us) server_negotiation.cc:407] Connection header received
0619 20:27:35.327256 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:35.327257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:35.327266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:35.327319 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:35.327681 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.328107 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.328735 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:35.328927 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:35.329009 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:35.330661 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:35.330662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:35.330662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:35.330669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:35.331825 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:35.332161 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:35.332162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:35.332163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:35.332213 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:35.332412 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:35.332413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:35.332413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:35.332506 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:35.332593 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:35.332700 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 20:27:35.332767 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:27:35.333375 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 27765
I20260619 20:27:35.338778 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 27901
I20260619 20:27:35.343242 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 28035
I20260619 20:27:35.347222 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 27670
2026-06-19T20:27:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1820 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[28187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[28187](info): set up 2 sockets
Jun 19 20:27:35 dist-test-slave-f9rk krb5kdc[28187](info): commencing operation
krb5kdc: starting...
W20260619 20:27:36.384984 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:36 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.62.keytab.
Entry for principal oryx/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.62.keytab.
I20260619 20:27:36.498620 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:38657
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.62.keytab
--principal=oryx/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:38657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:36.578269 28200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:36.578474 28200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:36.578496 28200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:36.579569 28200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:36.579602 28200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:36.579684 28200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:36.579715 28200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:36.579728 28200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:36.581192 28200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:38657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.62.keytab
--principal=oryx/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:38657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:36.581444 28200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:36.581700 28200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:36.586513 28206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:36.586519 28208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:36.586550 28205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:36.586802 28200 server_base.cc:1034] running on GCE node
I20260619 20:27:36.587034 28200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:36.587302 28200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:36.588442 28200 hybrid_clock.cc:648] HybridClock initialized: now 1781900856588431 us; error 33 us; skew 500 ppm
Jun 19 20:27:36 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:36.590865 28200 init.cc:377] Logged in from keytab as oryx/127.21.151.62@KRBTEST.COM (short username oryx)
I20260619 20:27:36.591274 28200 webserver.cc:460] Webserver started at http://127.21.151.62:39381/ using document root <none> and password file <none>
I20260619 20:27:36.591511 28200 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:36.591579 28200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:36.591753 28200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:36.592782 28200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "9b72a6e0b4b14623beafb8fbeb362cdc"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "322bc81c8e8f564f75e0462698ff2da5"
server_key_iv: "6c6c727b73db07efc14afc784a91bc2e"
server_key_version: "encryptionkey@0"
I20260619 20:27:36.593150 28200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "9b72a6e0b4b14623beafb8fbeb362cdc"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "322bc81c8e8f564f75e0462698ff2da5"
server_key_iv: "6c6c727b73db07efc14afc784a91bc2e"
server_key_version: "encryptionkey@0"
I20260619 20:27:36.594424 28200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:27:36.595118 28215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:36.595301 28200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:36.595394 28200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "9b72a6e0b4b14623beafb8fbeb362cdc"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "322bc81c8e8f564f75e0462698ff2da5"
server_key_iv: "6c6c727b73db07efc14afc784a91bc2e"
server_key_version: "encryptionkey@0"
I20260619 20:27:36.595479 28200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:36.619168 28200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:36.621631 28200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:36.621794 28200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:36.625630 28200 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:38657
I20260619 20:27:36.625672 28268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:38657 every 8 connection(s)
I20260619 20:27:36.626036 28200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:36.627007 28269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:36.629299 28269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc: Bootstrap starting.
I20260619 20:27:36.629868 28269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:36.630151 28269 log.cc:824] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:36.630851 28269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc: No bootstrap required, opened a new log
I20260619 20:27:36.632422 28269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } }
I20260619 20:27:36.632555 28269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:36.632577 28269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b72a6e0b4b14623beafb8fbeb362cdc, State: Initialized, Role: FOLLOWER
I20260619 20:27:36.632776 28269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } }
I20260619 20:27:36.632858 28269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:36.632903 28269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:36.632994 28269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:36.633008 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 28200
I20260619 20:27:36.633100 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:36.633299 22108 external_mini_cluster.cc:1428] Setting key 1801e236a4a57c655fca6c0cb2d5078f
I20260619 20:27:36.634070 28269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } }
I20260619 20:27:36.634174 28269 leader_election.cc:302] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b72a6e0b4b14623beafb8fbeb362cdc; no voters:
I20260619 20:27:36.634315 28269 leader_election.cc:288] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:36.634361 28274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:36.634543 28269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:36.634555 28274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 1 LEADER]: Becoming Leader. State: Replica: 9b72a6e0b4b14623beafb8fbeb362cdc, State: Running, Role: LEADER
I20260619 20:27:36.634662 28274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } }
I20260619 20:27:36.635162 28275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } }
I20260619 20:27:36.635246 28275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:36.635175 28276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b72a6e0b4b14623beafb8fbeb362cdc. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } }
I20260619 20:27:36.635286 28276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:36.635429 28278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:36.635985 28278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:36.637530 28278 catalog_manager.cc:1347] Generated new cluster ID: 6bedd11f191a42d7b3296864b84fd211
I20260619 20:27:36.637588 28278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:27:36 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:36.645450 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:36.633780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.21.151.62:38657)
0619 20:27:36.634028 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:36.634038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:36.634073 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:27:36.634185 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:36.634193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:36.634247 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:36.634399 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:36.635426 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:36.636750 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:36.637909 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:36.638489 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:36.638600 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:36.641313 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:36.641318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:36.641318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:36.641329 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:36.643916 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:36.644295 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:36.644297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:36.644298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:36.644409 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:36.644618 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:36.644619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:36.644619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:36.644881 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:36.645018 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:36.645134 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 20:27:36.645253 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":145,"threads_started":1}
I20260619 20:27:36.653017 28278 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:36.653505 28278 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:36.662878 28278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc: Generated new TSK 0
I20260619 20:27:36.663084 28278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab.
Entry for principal oryx/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab.
I20260619 20:27:36.690201 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38657
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab
--principal=oryx/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:36.769428 28295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:36.769634 28295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:36.769657 28295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:36.770738 28295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:36.770772 28295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:36.770798 28295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:36.772202 28295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab
--principal=oryx/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:36.772390 28295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:36.772558 28295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:36.777443 28303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:36.777448 28301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:36.777535 28300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:36.777575 28295 server_base.cc:1034] running on GCE node
I20260619 20:27:36.777812 28295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:36.778067 28295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:36.779227 28295 hybrid_clock.cc:648] HybridClock initialized: now 1781900856779214 us; error 28 us; skew 500 ppm
Jun 19 20:27:36 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:36.781742 28295 init.cc:377] Logged in from keytab as oryx/127.21.151.1@KRBTEST.COM (short username oryx)
I20260619 20:27:36.782212 28295 webserver.cc:460] Webserver started at http://127.21.151.1:36845/ using document root <none> and password file <none>
I20260619 20:27:36.782474 28295 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:36.782559 28295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:36.782707 28295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:36.783833 28295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "ad5b341bb7a44916b1943af42a35fd28"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "69c15973d256bd2bae2f3d4134c2af05"
server_key_iv: "c173e73678e36e282eca1bbaa0d67415"
server_key_version: "encryptionkey@0"
I20260619 20:27:36.784179 28295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "ad5b341bb7a44916b1943af42a35fd28"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "69c15973d256bd2bae2f3d4134c2af05"
server_key_iv: "c173e73678e36e282eca1bbaa0d67415"
server_key_version: "encryptionkey@0"
I20260619 20:27:36.785429 28295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:36.786118 28310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:36.786281 28295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:36.786381 28295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "ad5b341bb7a44916b1943af42a35fd28"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "69c15973d256bd2bae2f3d4134c2af05"
server_key_iv: "c173e73678e36e282eca1bbaa0d67415"
server_key_version: "encryptionkey@0"
I20260619 20:27:36.786448 28295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:36.813351 28295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:36.815680 28295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:36.815840 28295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:36.816083 28295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:36.816391 28295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:36.816444 28295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:36.816483 28295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:36.816517 28295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:36.821950 28295 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:36487
I20260619 20:27:36.822047 28424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:36487 every 8 connection(s)
I20260619 20:27:36.822345 28295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:27:36.825062 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 28295
I20260619 20:27:36.825166 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:36.825368 22108 external_mini_cluster.cc:1428] Setting key 43eb7359f87c97018405176b1ee8852f
Jun 19 20:27:36 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:36.832399 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:36.823040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:50347 (local address 127.21.151.62:38657)
0619 20:27:36.823103 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:36.823105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:36.823305 (+ 200us) server_negotiation.cc:407] Connection header received
0619 20:27:36.823774 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:36.823775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:36.823788 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:36.823839 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:36.824804 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:36.825638 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:36.826647 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:36.826885 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:36.826959 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:36.829325 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:36.829327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:36.829327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:36.829335 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:36.830987 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:36.831384 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:36.831385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:36.831385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:36.831450 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:36.831720 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:36.831721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:36.831721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:36.831866 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:36.831942 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:36.832197 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 20:27:36.832299 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:27:36.832417 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:36.823025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.1:50347)
0619 20:27:36.823182 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:36.823215 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:36.823689 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:36.823909 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:36.823912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:36.824497 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:36.824689 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:36.824694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:36.825741 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:36.825742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:36.826544 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:36.826546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:36.826966 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:36.826967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:36.827167 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:36.827683 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:36.827693 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:36.829236 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:36.831090 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:36.831096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:36.831097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:36.831312 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:36.831523 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:36.831524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:36.831524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:36.831660 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:36.831935 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:36.831941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:36.832058 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 20:27:36.832132 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:27:36.832249 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260619 20:27:36.833002 28425 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38657
I20260619 20:27:36.833124 28425 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:36.833290 28425 heartbeater.cc:511] Master 127.21.151.62:38657 requested a full tablet report, sending...
I20260619 20:27:36.833798 28232 ts_manager.cc:194] Registered new tserver with Master: ad5b341bb7a44916b1943af42a35fd28 (127.21.151.1:36487)
WARNING: no policy specified for oryx/127.21.151.2@KRBTEST.COM; defaulting to no policy
I20260619 20:27:36.834471 28232 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:50347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab.
Entry for principal oryx/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab.
I20260619 20:27:36.850546 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38657
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab
--principal=oryx/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:36.930464 28430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:36.930707 28430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:36.930748 28430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:36.931915 28430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:36.931980 28430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:36.932040 28430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:36.933497 28430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab
--principal=oryx/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:36.933786 28430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:36.934051 28430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:36.939630 28438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:36.939604 28436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:36.939631 28435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:36.939822 28430 server_base.cc:1034] running on GCE node
I20260619 20:27:36.940092 28430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:36.940330 28430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:36.941490 28430 hybrid_clock.cc:648] HybridClock initialized: now 1781900856941474 us; error 41 us; skew 500 ppm
Jun 19 20:27:36 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:36.943841 28430 init.cc:377] Logged in from keytab as oryx/127.21.151.2@KRBTEST.COM (short username oryx)
I20260619 20:27:36.944339 28430 webserver.cc:460] Webserver started at http://127.21.151.2:34819/ using document root <none> and password file <none>
I20260619 20:27:36.944573 28430 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:36.944641 28430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:36.944775 28430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:36.945784 28430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "e0721bde923c430dbeea0ad061fd32eb"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "5feb5a57a9e5ad26d54daee995449ec8"
server_key_iv: "4ecf2085d4edca347ab399e2222feb31"
server_key_version: "encryptionkey@0"
I20260619 20:27:36.946137 28430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "e0721bde923c430dbeea0ad061fd32eb"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "5feb5a57a9e5ad26d54daee995449ec8"
server_key_iv: "4ecf2085d4edca347ab399e2222feb31"
server_key_version: "encryptionkey@0"
I20260619 20:27:36.947415 28430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:36.948182 28446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:36.948359 28430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:36.948450 28430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "e0721bde923c430dbeea0ad061fd32eb"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "5feb5a57a9e5ad26d54daee995449ec8"
server_key_iv: "4ecf2085d4edca347ab399e2222feb31"
server_key_version: "encryptionkey@0"
I20260619 20:27:36.948534 28430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:36.979970 28430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:36.981891 28430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:36.982065 28430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:36.982326 28430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:36.982697 28430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:36.982730 28430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:36.982771 28430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:36.982800 28430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:36.988731 28430 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:33131
I20260619 20:27:36.988775 28559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:33131 every 8 connection(s)
I20260619 20:27:36.989113 28430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:27:36 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:36.996093 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 28430
I20260619 20:27:36.996207 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:36.996469 22108 external_mini_cluster.cc:1428] Setting key 75c1707d83cf870cff6784c3bf6eb4e2
I20260619 20:27:36.998412 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:36.989768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:52593 (local address 127.21.151.62:38657)
0619 20:27:36.989827 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:36.989829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:36.990071 (+ 242us) server_negotiation.cc:407] Connection header received
0619 20:27:36.990553 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:36.990554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:36.990590 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:36.990658 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:36.991551 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:36.992104 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:36.993113 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:36.993293 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:36.993343 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:36.995352 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:36.995353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:36.995353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:36.995360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:36.996873 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:36.997353 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:36.997354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:36.997354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:36.997407 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:36.997701 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:36.997702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:36.997703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:36.997857 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:36.997935 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:36.998177 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 20:27:36.998270 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:27:36.998474 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:36.989741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.2:52593)
0619 20:27:36.989886 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:36.989926 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:36.990467 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:36.990714 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:36.990717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:36.991282 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:36.991471 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:36.991476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:36.992178 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:36.992178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:36.993033 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:36.993035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:36.993362 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:36.993363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:36.993466 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:36.993904 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:36.993912 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:36.995267 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:36.996946 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:36.996951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:36.996953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:36.997277 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:36.997491 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:36.997493 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:36.997493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:36.997609 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:36.997924 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:36.997929 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:36.998050 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 20:27:36.998134 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:27:36.998291 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260619 20:27:36.999181 28560 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38657
I20260619 20:27:36.999292 28560 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:36.999483 28560 heartbeater.cc:511] Master 127.21.151.62:38657 requested a full tablet report, sending...
I20260619 20:27:36.999897 28232 ts_manager.cc:194] Registered new tserver with Master: e0721bde923c430dbeea0ad061fd32eb (127.21.151.2:33131)
I20260619 20:27:37.000273 28232 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:52593
WARNING: no policy specified for oryx/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab.
Entry for principal oryx/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab.
I20260619 20:27:37.021687 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38657
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab
--principal=oryx/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:37.102898 28565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:37.103120 28565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:37.103154 28565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:37.104426 28565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:37.104488 28565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:37.104534 28565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:37.105953 28565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab
--principal=oryx/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:37.106215 28565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:37.106470 28565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:37.111230 28571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:37.111402 28565 server_base.cc:1034] running on GCE node
W20260619 20:27:37.111236 28573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.111236 28570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:37.111775 28565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:37.112015 28565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:37.113150 28565 hybrid_clock.cc:648] HybridClock initialized: now 1781900857113136 us; error 38 us; skew 500 ppm
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:37.115695 28565 init.cc:377] Logged in from keytab as oryx/127.21.151.3@KRBTEST.COM (short username oryx)
I20260619 20:27:37.116101 28565 webserver.cc:460] Webserver started at http://127.21.151.3:46493/ using document root <none> and password file <none>
I20260619 20:27:37.116313 28565 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:37.116357 28565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:37.116494 28565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:37.117518 28565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "d1cbe361770b455492864ccae615cf28"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "6939c9e1197e3052fcc9292a2b2c25fe"
server_key_iv: "8654220d6bea2b959a15a407a4d3e713"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.117872 28565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "d1cbe361770b455492864ccae615cf28"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "6939c9e1197e3052fcc9292a2b2c25fe"
server_key_iv: "8654220d6bea2b959a15a407a4d3e713"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.119117 28565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:27:37.119801 28580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.119961 28565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:37.120056 28565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "d1cbe361770b455492864ccae615cf28"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "6939c9e1197e3052fcc9292a2b2c25fe"
server_key_iv: "8654220d6bea2b959a15a407a4d3e713"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.120132 28565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:37.135579 28565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:37.137812 28565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:37.137964 28565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:37.138190 28565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:37.138478 28565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:37.138530 28565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.138567 28565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:37.138595 28565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.144132 28565 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:44885
I20260619 20:27:37.144176 28693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:44885 every 8 connection(s)
I20260619 20:27:37.144459 28565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:37.145977 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 28565
I20260619 20:27:37.146072 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:37.146250 22108 external_mini_cluster.cc:1428] Setting key 4313e3cb33541a78d6e3030001060fd4
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:37.153347 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.145101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:35965 (local address 127.21.151.62:38657)
0619 20:27:37.145140 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.145142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.145393 (+ 251us) server_negotiation.cc:407] Connection header received
0619 20:27:37.145888 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.145889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.145901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.145962 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.146935 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.147409 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.148456 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.148646 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.148726 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.150697 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.150699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.150699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.150707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.151982 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.152401 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.152402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.152403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.152461 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.152723 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.152724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.152724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.152835 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.152911 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.153176 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.153257 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:27:37.153524 28696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.145097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.3:35965)
0619 20:27:37.145265 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.145320 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.145789 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.146053 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.146057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.146668 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.146843 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.146847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.147504 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.147504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.148355 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.148357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.148735 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.148736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.148864 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.149272 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.149279 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.150603 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.152083 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.152090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.152092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.152311 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.152544 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.152545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.152546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.152655 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.152902 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.152908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.153048 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.153155 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.153304 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260619 20:27:37.153967 28694 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38657
I20260619 20:27:37.154067 28694 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:37.154232 28694 heartbeater.cc:511] Master 127.21.151.62:38657 requested a full tablet report, sending...
I20260619 20:27:37.154590 28232 ts_manager.cc:194] Registered new tserver with Master: d1cbe361770b455492864ccae615cf28 (127.21.151.3:44885)
I20260619 20:27:37.154918 28232 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:35965
I20260619 20:27:37.157829 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.151.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.61.keytab.
Entry for principal oryx/127.21.151.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.61.keytab.
I20260619 20:27:37.180173 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.61:37133
--webserver_interface=127.21.151.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.61.keytab
--principal=oryx/127.21.151.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:38657,127.21.151.61:37133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:37.258792 28699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:37.258987 28699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:37.259009 28699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:37.260095 28699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:37.260130 28699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:37.260144 28699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:37.260156 28699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:37.260169 28699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:37.261551 28699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:38657,127.21.151.61:37133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.61.keytab
--principal=oryx/127.21.151.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.61:37133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.151.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
W20260619 20:27:37.261654 28699 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.151.62:38657,127.21.151.61:37133'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 20:27:37.261760 28699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:37.261925 28699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:37.266498 28708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.266498 28706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.266532 28705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:37.266701 28699 server_base.cc:1034] running on GCE node
I20260619 20:27:37.266953 28699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:37.267179 28699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:37.268325 28699 hybrid_clock.cc:648] HybridClock initialized: now 1781900857268314 us; error 27 us; skew 500 ppm
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:37.270751 28699 init.cc:377] Logged in from keytab as oryx/127.21.151.61@KRBTEST.COM (short username oryx)
I20260619 20:27:37.271211 28699 webserver.cc:460] Webserver started at http://127.21.151.61:33045/ using document root <none> and password file <none>
I20260619 20:27:37.271457 28699 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:37.271525 28699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:37.271703 28699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:37.272727 28699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data/instance:
uuid: "a34fef0e27f1436fb3af99ad4c4baeaf"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "8e304739189ff5d2b524d362c94069ae"
server_key_iv: "045d09252e415b44ae759ff4be2850c0"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.273082 28699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal/instance:
uuid: "a34fef0e27f1436fb3af99ad4c4baeaf"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "8e304739189ff5d2b524d362c94069ae"
server_key_iv: "045d09252e415b44ae759ff4be2850c0"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.274344 28699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:37.275022 28715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.275180 28699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:37.275285 28699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal
uuid: "a34fef0e27f1436fb3af99ad4c4baeaf"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "8e304739189ff5d2b524d362c94069ae"
server_key_iv: "045d09252e415b44ae759ff4be2850c0"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.275357 28699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:37.283274 28699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:37.285488 28699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:37.285647 28699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:37.289664 28699 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.61:37133
I20260619 20:27:37.289746 28767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.61:37133 every 8 connection(s)
I20260619 20:27:37.290067 28699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
I20260619 20:27:37.290958 28768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:37.292435 28768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:27:37.294587 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 28699
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.61@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.151.61@KRBTEST.COM
I20260619 20:27:37.302188 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.293107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.21.151.62:38657)
0619 20:27:37.293159 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.293161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.293374 (+ 213us) server_negotiation.cc:407] Connection header received
0619 20:27:37.293798 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.293799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.293808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.293855 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.294435 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.295225 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.296662 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.296869 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.296955 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.299092 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.299093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.299094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.299103 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.300738 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.301150 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.301151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.301152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.301202 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.301453 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.301454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.301455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.301641 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.301701 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.302047 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.302125 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:37.302500 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.293099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39328)
0619 20:27:37.293259 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.293293 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.293720 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.293912 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.293916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.294177 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.294357 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.294361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.295338 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.295339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.296574 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.296577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.296963 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.296964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.297084 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.297567 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.297574 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.298976 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.300809 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.300815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.300817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.301075 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.301267 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.301268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.301268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.301383 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.301714 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.301720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.301882 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.302012 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.302184 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260619 20:27:37.303143 28768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:27:37.304207 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.295139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.21.151.61:37133)
0619 20:27:37.295299 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.295303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.295352 (+ 49us) server_negotiation.cc:407] Connection header received
0619 20:27:37.295420 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.295422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.295438 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.295501 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.296239 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.296904 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.297837 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.298267 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.298352 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.301053 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.301059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.301059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.301066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.302912 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.303324 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.303328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.303328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.303404 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.303724 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.303724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.303725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.303889 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.304049 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.304093 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.304147 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":108,"threads_started":1}
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.61@KRBTEST.COM for oryx/127.21.151.61@KRBTEST.COM
I20260619 20:27:37.309599 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.303388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:37133 (local address 127.0.0.1:60206)
0619 20:27:37.303416 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.303426 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.303459 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.303681 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.303682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.303821 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.303850 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.303852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.304785 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.304785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.305294 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.305295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.305613 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.305613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.305653 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.305960 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.305965 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.306999 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.308867 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.308868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.308868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.309028 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.309193 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.309193 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.309193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.309240 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.309401 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.309401 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.309432 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.309476 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.309544 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260619 20:27:37.309711 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.303417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.21.151.61:37133)
0619 20:27:37.303540 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.303542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.303548 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:27:37.303564 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.303565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.303574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.303664 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.303937 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.304626 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.305358 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.305551 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.305602 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.307083 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.307084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.307084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.307090 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.308801 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.309089 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.309090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.309091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.309138 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.309276 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.309276 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.309276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.309355 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.309399 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.309499 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.309586 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260619 20:27:37.310835 28768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Bootstrap starting.
I20260619 20:27:37.311439 28768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:37.311770 28768 log.cc:824] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:37.312422 28768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: No bootstrap required, opened a new log
I20260619 20:27:37.313736 28768 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } }
I20260619 20:27:37.313850 28768 raft_consensus.cc:374] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:37.313894 28768 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34fef0e27f1436fb3af99ad4c4baeaf, State: Initialized, Role: FOLLOWER
I20260619 20:27:37.314013 28768 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } }
I20260619 20:27:37.314245 28775 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } } }
I20260619 20:27:37.314306 28768 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:37.314321 28775 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:27:37.315879 28786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:27:37.315939 28786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:27:37.317355 28699 master_runner.cc:428] Detected that this master a34fef0e27f1436fb3af99ad4c4baeaf is joining an existing cluster
I20260619 20:27:37.317394 28699 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.151.61:37133
I20260619 20:27:37.324941 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.318351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:37133 (local address 127.0.0.1:60208)
0619 20:27:37.318634 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.318641 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.318710 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.318819 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.318820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.318992 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.319055 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.319057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.319650 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.319651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.320425 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.320427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.320937 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.320938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.321009 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.321456 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.321462 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.321863 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.323773 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.323774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.323775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.323993 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.324215 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.324216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.324217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.324283 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.324576 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.324577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.324620 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.324712 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.324807 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":74,"threads_started":1}
I20260619 20:27:37.325048 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.318508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.21.151.61:37133)
0619 20:27:37.318539 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.318541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.318694 (+ 153us) server_negotiation.cc:407] Connection header received
0619 20:27:37.318759 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.318760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.318769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.318817 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.319108 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.319561 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.320524 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.320841 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.320916 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.321958 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.321960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.321960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.321967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.323678 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.324071 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.324072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.324072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.324137 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.324356 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.324358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.324358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.324487 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.324576 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.324736 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.324965 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":17}
I20260619 20:27:37.325970 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.318358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.21.151.62:38657)
0619 20:27:37.318400 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.318402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.318539 (+ 137us) server_negotiation.cc:407] Connection header received
0619 20:27:37.318603 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.318604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.318617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.318677 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.319077 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.319849 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.320732 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.320996 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.321064 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.322432 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.322433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.322433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.322440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.324335 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.324683 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.324684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.324684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.324746 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.324919 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.324920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.324920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.325526 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.325594 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.325815 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.325887 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:37.325979 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.318315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39330)
0619 20:27:37.318476 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.318487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.318549 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.318694 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.318695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.318903 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.318994 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.318996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.319928 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.319928 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.320657 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.320659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.321078 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.321079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.321145 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.321647 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.321652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.322112 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.324399 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.324400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.324401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.324608 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.324799 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.324799 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.324799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.324854 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.325582 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.325583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.325623 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.325770 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.325868 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.62@KRBTEST.COM for oryx/127.21.151.61@KRBTEST.COM
I20260619 20:27:37.335498 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.327924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:37133 (local address 127.0.0.1:60214)
0619 20:27:37.328101 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.328149 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.328222 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.328365 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.328368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.328698 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.328750 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.328753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.329312 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.329313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.330241 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.330243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.330555 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.330556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.330643 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.331028 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.331032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.332814 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.334457 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.334463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.334464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.334649 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.334858 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.334859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.334860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.334925 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.335216 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.335222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.335263 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.335328 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.335413 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260619 20:27:37.335537 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.327999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.21.151.61:37133)
0619 20:27:37.328037 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.328039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.328200 (+ 161us) server_negotiation.cc:407] Connection header received
0619 20:27:37.328308 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.328309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.328317 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.328359 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.328821 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.329240 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.330320 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.330487 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.330530 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.332908 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.332910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.332910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.332916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.334377 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.334719 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.334720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.334720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.334774 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.335030 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.335031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.335031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.335151 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.335219 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.335370 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.335443 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:37.336458 28232 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.151.61:37133: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf"
member_type: NON_VOTER
last_known_addr {
host: "127.21.151.61"
port: 37133
}
attrs {
promote: true
}
}
dest_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc"
cas_config_opid_index: -1
I20260619 20:27:37.336782 28232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } }
I20260619 20:27:37.337244 28796 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a34fef0e27f1436fb3af99ad4c4baeaf (127.21.151.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } } }
I20260619 20:27:37.337545 28742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:37.338151 28796 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.151.61:37133
I20260619 20:27:37.338241 28796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } } }
I20260619 20:27:37.338312 28796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:37.338387 28699 master.cc:537] Master@127.21.151.61:37133 shutting down...
I20260619 20:27:37.338445 28742 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Refusing update from remote peer 9b72a6e0b4b14623beafb8fbeb362cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 20:27:37.338748 28795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 20:27:37.340389 28217 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 20:27:37.340909 28217 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc -> Peer a34fef0e27f1436fb3af99ad4c4baeaf (127.21.151.61:37133): Couldn't send request to peer a34fef0e27f1436fb3af99ad4c4baeaf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 20:27:37.341080 28699 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:37.341188 28699 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:37.341241 28699 tablet_replica.cc:330] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: stopping tablet replica
I20260619 20:27:37.353605 28699 master.cc:552] Master@127.21.151.61:37133 shutdown complete.
I20260619 20:27:37.353652 28699 master_runner.cc:305] Clearing existing system tablet
I20260619 20:27:37.353806 28699 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:37.353843 28699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:37.354460 28699 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260619 20:27:37.354954 28800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.355087 28699 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:37.355162 28699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.355191 28699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal
uuid: "a34fef0e27f1436fb3af99ad4c4baeaf"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "8e304739189ff5d2b524d362c94069ae"
server_key_iv: "045d09252e415b44ae759ff4be2850c0"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.355943 28699 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:37.357210 28699 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 20:27:37.357290 28699 log.cc:1196] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 20:27:37.357486 28699 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Deleting consensus metadata
I20260619 20:27:37.357621 28699 master_runner.cc:315] Copying system tablet from 127.21.151.62:38657
I20260619 20:27:37.358222 28699 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: tablet copy: Beginning tablet copy session from remote peer at address 127.21.151.62:38657
I20260619 20:27:37.364406 28807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.358523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39340)
0619 20:27:37.358640 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.358651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.358715 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.358858 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.358859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.359005 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.359057 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.359059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.359998 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.359998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.360541 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.360543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.361016 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.361017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.361063 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.361533 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.361538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.361832 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.363479 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.363480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.363481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.363688 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.363872 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.363873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.363873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.363922 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.364177 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.364178 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.364212 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.364264 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.364336 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260619 20:27:37.364452 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.358577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.21.151.62:38657)
0619 20:27:37.358629 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.358631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.358719 (+ 88us) server_negotiation.cc:407] Connection header received
0619 20:27:37.358782 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.358783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.358794 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.358854 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.359121 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.359924 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.360630 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.360938 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.361003 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.361915 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.361916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.361916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.361923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.363398 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.363745 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.363746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.363746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.363808 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.363982 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.363983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.363983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.364116 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.364183 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.364286 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.364356 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:27:37.364730 28253 tablet_copy_service.cc:140] P 9b72a6e0b4b14623beafb8fbeb362cdc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a34fef0e27f1436fb3af99ad4c4baeaf ({username='oryx', principal='oryx/127.21.151.61@KRBTEST.COM'} at 127.0.0.1:39340)
I20260619 20:27:37.364830 28253 tablet_copy_service.cc:161] P 9b72a6e0b4b14623beafb8fbeb362cdc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a34fef0e27f1436fb3af99ad4c4baeaf at {username='oryx', principal='oryx/127.21.151.61@KRBTEST.COM'} at 127.0.0.1:39340: session id = a34fef0e27f1436fb3af99ad4c4baeaf-00000000000000000000000000000000
I20260619 20:27:37.365442 28253 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 20:27:37.366048 28699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 20:27:37.367256 28699 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: tablet copy: Starting download of 0 data blocks...
I20260619 20:27:37.367401 28699 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: tablet copy: Starting download of 1 WAL segments...
I20260619 20:27:37.368260 28699 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 20:27:37.369153 28253 tablet_copy_service.cc:342] P 9b72a6e0b4b14623beafb8fbeb362cdc: Request end of tablet copy session a34fef0e27f1436fb3af99ad4c4baeaf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.151.61@KRBTEST.COM'} at 127.0.0.1:39340
I20260619 20:27:37.369212 28253 tablet_copy_service.cc:433] P 9b72a6e0b4b14623beafb8fbeb362cdc: ending tablet copy session a34fef0e27f1436fb3af99ad4c4baeaf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a34fef0e27f1436fb3af99ad4c4baeaf
I20260619 20:27:37.371376 28699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:37.372478 28809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.372617 28808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.372665 28811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:37.373212 28699 server_base.cc:1034] running on GCE node
I20260619 20:27:37.373308 28699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:37.373430 28699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:37.374557 28699 hybrid_clock.cc:648] HybridClock initialized: now 1781900857374548 us; error 24 us; skew 500 ppm
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:37.375958 28699 init.cc:377] Logged in from keytab as oryx/127.21.151.61@KRBTEST.COM (short username oryx)
I20260619 20:27:37.376250 28699 webserver.cc:460] Webserver started at http://127.21.151.61:45805/ using document root <none> and password file <none>
I20260619 20:27:37.376367 28699 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:37.376420 28699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:37.376998 28699 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:37.377493 28818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.377643 28699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:37.377701 28699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal
uuid: "a34fef0e27f1436fb3af99ad4c4baeaf"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "8e304739189ff5d2b524d362c94069ae"
server_key_iv: "045d09252e415b44ae759ff4be2850c0"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.377758 28699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:37.389097 28699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:37.390659 28699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:37.393733 28699 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.61:37133
I20260619 20:27:37.394155 28699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
I20260619 20:27:37.394140 28870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.61:37133 every 8 connection(s)
I20260619 20:27:37.394851 28871 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:27:37.395290 28871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Bootstrap starting.
I20260619 20:27:37.397564 28871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:27:37.397842 28871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Bootstrap complete.
I20260619 20:27:37.398053 28871 raft_consensus.cc:348] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } }
I20260619 20:27:37.398139 28871 raft_consensus.cc:729] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a34fef0e27f1436fb3af99ad4c4baeaf, State: Initialized, Role: LEARNER
I20260619 20:27:37.398180 28871 consensus_queue.cc:260] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } }
I20260619 20:27:37.398342 28871 sys_catalog.cc:564] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:37.398335 28874 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } } }
I20260619 20:27:37.398423 28874 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: This master's current role is: LEARNER
I20260619 20:27:37.399748 28885 catalog_manager.cc:1259] Loaded cluster ID: 6bedd11f191a42d7b3296864b84fd211
I20260619 20:27:37.399807 28885 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: loading cluster ID for follower catalog manager: success
I20260619 20:27:37.400857 28885 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: acquiring CA information for follower catalog manager: success
I20260619 20:27:37.401145 28885 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:27:37.412209 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.405753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.21.151.61:37133)
0619 20:27:37.405866 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.405868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.405878 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:37.405902 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.405902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.405911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.405998 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.406518 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.407158 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.407938 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.408140 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.408245 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.409509 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.409510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.409511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.409518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.411001 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.411375 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.411377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.411377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.411438 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.411670 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.411671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.411671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.411780 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.411891 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.412002 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.412082 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
WARNING: no policy specified for oryx/127.21.151.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.60.keytab.
Entry for principal oryx/127.21.151.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.60.keytab.
I20260619 20:27:37.436592 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.60:35461
--webserver_interface=127.21.151.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.60.keytab
--principal=oryx/127.21.151.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:38657,127.21.151.61:37133,127.21.151.60:35461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:37.517768 28889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:37.517961 28889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:37.517982 28889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:37.519084 28889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:37.519125 28889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:37.519140 28889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:37.519176 28889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:37.519201 28889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:37.520735 28889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:38657,127.21.151.61:37133,127.21.151.60:35461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.60.keytab
--principal=oryx/127.21.151.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.60:35461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.151.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:37.520984 28889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:37.521245 28889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:37.525923 28898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.525934 28895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.525918 28894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:37.526176 28889 server_base.cc:1034] running on GCE node
I20260619 20:27:37.526387 28889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:37.526614 28889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:37.527731 28889 hybrid_clock.cc:648] HybridClock initialized: now 1781900857527721 us; error 26 us; skew 500 ppm
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:37.530053 28889 init.cc:377] Logged in from keytab as oryx/127.21.151.60@KRBTEST.COM (short username oryx)
I20260619 20:27:37.530464 28889 webserver.cc:460] Webserver started at http://127.21.151.60:46507/ using document root <none> and password file <none>
I20260619 20:27:37.530692 28889 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:37.530757 28889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:37.530922 28889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:37.532034 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data/instance:
uuid: "b187e099fc03474aaa6724817eda7ec6"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "897774d1d871d21a4e010194ac55605a"
server_key_iv: "0f38edd0deb2fc6af21cfea603afe85e"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.532389 28889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal/instance:
uuid: "b187e099fc03474aaa6724817eda7ec6"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "897774d1d871d21a4e010194ac55605a"
server_key_iv: "0f38edd0deb2fc6af21cfea603afe85e"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.533656 28889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:37.534358 28905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.534543 28889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:37.534638 28889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal
uuid: "b187e099fc03474aaa6724817eda7ec6"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "897774d1d871d21a4e010194ac55605a"
server_key_iv: "0f38edd0deb2fc6af21cfea603afe85e"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.534725 28889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:37.556371 28889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:37.558182 28889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:37.558347 28889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:37.562199 28889 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.60:35461
I20260619 20:27:37.562234 28957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.60:35461 every 8 connection(s)
I20260619 20:27:37.562613 28889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
I20260619 20:27:37.563592 28958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:37.565152 28958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.60@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:37.571554 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 28889
I20260619 20:27:37.573871 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.565925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.21.151.62:38657)
0619 20:27:37.565981 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.565983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.566155 (+ 172us) server_negotiation.cc:407] Connection header received
0619 20:27:37.566591 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.566592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.566602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.566645 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.567227 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.567775 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.568682 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.568856 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.568903 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.570994 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.570995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.570996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.571002 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.572476 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.572886 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.572887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.572887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.572944 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.573229 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.573230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.573230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.573342 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.573425 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.573690 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.573782 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:27:37.574043 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.565787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39344)
0619 20:27:37.566035 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.566067 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.566497 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.566719 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.566722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.566971 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.567157 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.567161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.567847 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.567848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.568614 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.568616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.568935 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.568935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.569041 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.569547 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.569555 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.570916 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.572578 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.572585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.572587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.572798 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.573031 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.573031 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.573032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.573157 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.573426 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.573432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.573560 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.573655 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.573820 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
I20260619 20:27:37.574545 28958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.151.60@KRBTEST.COM
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.60@KRBTEST.COM for oryx/127.21.151.61@KRBTEST.COM
I20260619 20:27:37.580807 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.572124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.21.151.60:35461)
0619 20:27:37.572270 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.572275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.572352 (+ 77us) server_negotiation.cc:407] Connection header received
0619 20:27:37.572416 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.572418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.572431 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.572497 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.573059 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.573735 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.574752 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.575220 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.575309 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.577818 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.577823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.577824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.577829 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.579563 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.579982 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.579985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.579985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.580035 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.580229 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.580230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.580230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.580345 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.580491 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.580573 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.580679 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260619 20:27:37.581574 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.574852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:37133 (local address 127.0.0.1:60244)
0619 20:27:37.574900 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.574910 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.574951 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.575093 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.575093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.575193 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.575267 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.575268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.575995 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.575996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.576501 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.576502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.576964 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.576965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.577019 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.577414 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.577419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.578636 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.580361 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.580362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.580363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.580517 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.580735 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.580735 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.580736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.580790 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.581252 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.581253 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.581290 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.581386 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.581477 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:27:37.581674 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.574930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.21.151.61:37133)
0619 20:27:37.574994 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.574995 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.575004 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:37.575025 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.575026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.575038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.575093 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.575339 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.575878 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.576593 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.576884 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.576951 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.578730 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.578731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.578732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.578738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.580287 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.580582 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.580583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.580583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.580672 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.580987 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.580988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.580988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.581159 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.581219 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.581453 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.581566 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:27:37.582021 28958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.60@KRBTEST.COM for oryx/127.21.151.60@KRBTEST.COM
I20260619 20:27:37.587962 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.582273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:35461 (local address 127.0.0.1:57268)
0619 20:27:37.582320 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.582330 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.582365 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.582469 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.582470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.582597 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.582619 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.582620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.583302 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.583303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.584136 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.584138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.584528 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.584529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.584566 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.584852 (+ 286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.584855 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.585881 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.587031 (+ 1150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.587032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.587032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.587177 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.587352 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.587353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.587353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.587399 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.587558 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.587558 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.587588 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.587708 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.587830 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:27:37.588022 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.582343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57268 (local address 127.21.151.60:35461)
0619 20:27:37.582375 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.582376 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.582386 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:37.582405 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.582406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.582415 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.582467 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.582695 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.583181 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.584208 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.584446 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.584504 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.585952 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.585953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.585953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.585958 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.586968 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.587255 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.587255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.587256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.587299 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.587435 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.587435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.587436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.587512 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.587554 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.587745 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.587881 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":18}
I20260619 20:27:37.589265 28958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: Bootstrap starting.
I20260619 20:27:37.590206 28958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:37.590500 28958 log.cc:824] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:37.591217 28958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: No bootstrap required, opened a new log
I20260619 20:27:37.592729 28958 raft_consensus.cc:348] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } }
I20260619 20:27:37.592850 28958 raft_consensus.cc:374] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:37.592931 28958 raft_consensus.cc:729] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b187e099fc03474aaa6724817eda7ec6, State: Initialized, Role: FOLLOWER
I20260619 20:27:37.593092 28958 consensus_queue.cc:260] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } }
I20260619 20:27:37.593345 28963 sys_catalog.cc:455] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } } }
I20260619 20:27:37.593389 28958 sys_catalog.cc:564] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:37.593457 28963 sys_catalog.cc:458] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:27:37.595013 28974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:27:37.595077 28974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:27:37.597272 28889 master_runner.cc:428] Detected that this master b187e099fc03474aaa6724817eda7ec6 is joining an existing cluster
I20260619 20:27:37.597318 28889 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.151.60:35461
I20260619 20:27:37.604820 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.598344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.21.151.61:37133)
0619 20:27:37.598390 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.598392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.598530 (+ 138us) server_negotiation.cc:407] Connection header received
0619 20:27:37.598696 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.598697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.598709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.598761 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.599192 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.600065 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.601164 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.601455 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.601492 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.602441 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.602442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.602443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.602450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.603750 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.604092 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.604094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.604094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.604146 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.604331 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.604332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.604333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.604440 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.604515 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.604657 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.604728 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:37.604835 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.598312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:37133 (local address 127.0.0.1:60260)
0619 20:27:37.598464 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.598475 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.598633 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.598807 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.598808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.599023 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.599106 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.599108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.600164 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.600164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.601075 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.601077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.601571 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.601572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.601631 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.602040 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.602044 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.602355 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.603830 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.603831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.603831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.604012 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.604210 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.604211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.604211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.604263 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.604500 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.604500 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.604551 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.604635 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.604729 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260619 20:27:37.628135 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.598563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.21.151.60:35461)
0619 20:27:37.598612 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.598613 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.598972 (+ 359us) server_negotiation.cc:407] Connection header received
0619 20:27:37.599040 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.599040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.599050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.599105 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.599595 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.600328 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.601354 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.601625 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.601663 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.625318 (+ 23655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.625320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.625320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.625327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.626922 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.627326 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.627327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.627328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.627372 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.627535 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.627536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.627536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.627699 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.627741 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.627905 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.628016 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:27:37.628379 28981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.598550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:35461 (local address 127.0.0.1:57282)
0619 20:27:37.598901 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.598928 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.598991 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.599150 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.599151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.599408 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.599512 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.599514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.600428 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.600429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.601266 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.601269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.601728 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.601728 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.601788 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.624274 (+ 22486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.624280 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.624899 (+ 619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.627023 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.627024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.627024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.627241 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.627430 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.627431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.627431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.627478 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.627780 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.627781 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.627814 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.627992 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.628083 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":72,"threads_started":1}
I20260619 20:27:37.629254 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.598788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.21.151.62:38657)
0619 20:27:37.598828 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.598830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.599119 (+ 289us) server_negotiation.cc:407] Connection header received
0619 20:27:37.599211 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.599211 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.599222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.599250 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.599690 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.600423 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.601439 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.601732 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.601773 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.625021 (+ 23248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.625023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.625024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.625034 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.627691 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.628392 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.628393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.628394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.628456 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.628758 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.628760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.628760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.628915 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.628969 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.629135 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.629200 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:37.629314 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.598731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39352)
0619 20:27:37.599071 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.599081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.599141 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.599304 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.599305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.599524 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.599577 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.599579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.600516 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.600517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.601358 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.601359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.601850 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.601851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.601895 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.624348 (+ 22453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.624353 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.624899 (+ 546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.627780 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.627781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.627782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.628154 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.628631 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.628632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.628632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.628689 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.628985 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.628986 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.629033 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.629108 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.629210 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":111,"threads_started":1}
I20260619 20:27:37.632143 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 28295
I20260619 20:27:37.638652 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:36487
--local_ip_for_outbound_sockets=127.21.151.1
--tserver_master_addrs=127.21.151.62:38657
--webserver_port=36845
--webserver_interface=127.21.151.1
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab
--principal=oryx/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.151.62:38657,127.21.151.61:37133
--tserver_master_addrs=127.21.151.62:38657,127.21.151.61:37133,127.21.151.60:35461 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900856, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.62@KRBTEST.COM for oryx/127.21.151.60@KRBTEST.COM
I20260619 20:27:37.643676 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.631308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.21.151.60:35461)
0619 20:27:37.631354 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.631356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.631363 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:27:37.631433 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.631433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.631441 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.631507 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.631827 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.632513 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.638339 (+ 5826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.638563 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.638653 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.640676 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.640677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.640677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.640684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.642413 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.642827 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.642829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.642829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.642886 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.643116 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.643117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.643117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.643240 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.643302 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.643480 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.643555 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 20:27:37.643707 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.631259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:35461 (local address 127.0.0.1:57294)
0619 20:27:37.631305 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.631317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.631362 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.631521 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.631522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.631714 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.631762 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.631764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.637669 (+ 5905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.637670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.638250 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.638253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.638654 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.638655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.638723 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.639123 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.639128 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.640558 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.642499 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.642500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.642501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.642739 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.642962 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.642963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.642964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.643043 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.643309 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.643310 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.643361 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.643432 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.643542 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 20:27:37.644641 28232 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.151.60:35461: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b187e099fc03474aaa6724817eda7ec6"
member_type: NON_VOTER
last_known_addr {
host: "127.21.151.60"
port: 35461
}
attrs {
promote: true
}
}
dest_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc"
cas_config_opid_index: 5
I20260619 20:27:37.644873 28232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } }
I20260619 20:27:37.645447 28932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:37.645479 28796 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b187e099fc03474aaa6724817eda7ec6 (127.21.151.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.646402 28796 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.151.60:35461
I20260619 20:27:37.646472 28796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.646543 28796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:37.646559 28932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 FOLLOWER]: Refusing update from remote peer 9b72a6e0b4b14623beafb8fbeb362cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 20:27:37.646822 28795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:37.647087 28889 master.cc:537] Master@127.21.151.60:35461 shutting down...
W20260619 20:27:37.647476 28217 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc -> Peer b187e099fc03474aaa6724817eda7ec6 (127.21.151.60:35461): Couldn't send request to peer b187e099fc03474aaa6724817eda7ec6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 20:27:37.650516 28889 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:37.650599 28889 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:37.650638 28889 tablet_replica.cc:330] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: stopping tablet replica
I20260619 20:27:37.651826 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.645412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60276 (local address 127.21.151.61:37133)
0619 20:27:37.645450 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.645452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.647774 (+ 2322us) server_negotiation.cc:407] Connection header received
0619 20:27:37.647890 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.647891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.647916 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.647960 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:37.648185 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.648970 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.650548 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.651130 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.651193 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.651282 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.651719 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.651759 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:27:37.652926 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.645320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:37133 (local address 127.0.0.1:60276)
0619 20:27:37.647680 (+ 2360us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.647692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.647810 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.648057 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.648058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.648060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:37.648098 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.648100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.649164 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.649164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.650447 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.650450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.651526 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.651527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.651596 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.651639 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.652820 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.652862 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2343}
I20260619 20:27:37.653301 28845 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 LEARNER]: Refusing update from remote peer 9b72a6e0b4b14623beafb8fbeb362cdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 20:27:37.653533 28795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 20:27:37.654198 28889 master.cc:552] Master@127.21.151.60:35461 shutdown complete.
I20260619 20:27:37.654232 28889 master_runner.cc:305] Clearing existing system tablet
I20260619 20:27:37.654146 28845 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b187e099fc03474aaa6724817eda7ec6 (127.21.151.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.654358 28889 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:37.654397 28889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:37.655162 28874 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b72a6e0b4b14623beafb8fbeb362cdc. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.655236 28874 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: This master's current role is: LEARNER
I20260619 20:27:37.655179 28986 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: NON_VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: true } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.655274 28986 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: This master's current role is: LEARNER
I20260619 20:27:37.655838 28795 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc: attempting to promote NON_VOTER a34fef0e27f1436fb3af99ad4c4baeaf to VOTER
I20260619 20:27:37.656026 28795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } }
I20260619 20:27:37.656625 28845 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 LEARNER]: Refusing update from remote peer 9b72a6e0b4b14623beafb8fbeb362cdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 20:27:37.656850 28987 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 20:27:37.656896 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.656492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:35461 (local address 127.0.0.1:57308)
0619 20:27:37.656831 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.656871 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326}
I20260619 20:27:37.657225 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.657153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:35461 (local address 127.0.0.1:57310)
0619 20:27:37.657182 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.657202 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260619 20:27:37.657289 28217 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc -> Peer b187e099fc03474aaa6724817eda7ec6 (127.21.151.60:35461): Couldn't send request to peer b187e099fc03474aaa6724817eda7ec6. Status: Network error: Client connection negotiation failed: client connection to 127.21.151.60:35461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 20:27:37.657667 28795 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a34fef0e27f1436fb3af99ad4c4baeaf (127.21.151.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.658504 28795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.658569 28795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:37.658537 28987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.658620 28987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:37.658810 28845 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a34fef0e27f1436fb3af99ad4c4baeaf (127.21.151.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.659606 28986 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.659727 28986 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:27:37.660005 28889 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.001s
I20260619 20:27:37.660514 28988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.660672 28889 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:37.660727 28889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.660750 28889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal
uuid: "b187e099fc03474aaa6724817eda7ec6"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "897774d1d871d21a4e010194ac55605a"
server_key_iv: "0f38edd0deb2fc6af21cfea603afe85e"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.661460 28889 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:37.662727 28889 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 20:27:37.662789 28889 log.cc:1196] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 20:27:37.662971 28889 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: Deleting consensus metadata
I20260619 20:27:37.663287 28889 master_runner.cc:315] Copying system tablet from 127.21.151.62:38657
I20260619 20:27:37.663848 28889 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: tablet copy: Beginning tablet copy session from remote peer at address 127.21.151.62:38657
I20260619 20:27:37.670729 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.664123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.21.151.62:38657)
0619 20:27:37.664161 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.664163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.664329 (+ 166us) server_negotiation.cc:407] Connection header received
0619 20:27:37.664351 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.664352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.664362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.664429 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.664755 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.665475 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.666426 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.666714 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.666775 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.667899 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.667901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.667901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.667907 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.669534 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.669925 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.669926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.669927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.669986 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.670207 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.670208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.670208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.670348 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.670414 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.670583 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.670653 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:27:37.670732 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.664092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39368)
0619 20:27:37.664211 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.664222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.664277 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.664441 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.664442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.664628 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.664678 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.664680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.665560 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.665581 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.666341 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.666343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.666775 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.666776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.666840 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.667305 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.667310 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.667818 (+ 508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.669620 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.669621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.669621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.669844 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.670067 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.670068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.670068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.670135 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.670420 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.670421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.670467 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.670544 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.670642 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260619 20:27:37.671305 28253 tablet_copy_service.cc:140] P 9b72a6e0b4b14623beafb8fbeb362cdc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b187e099fc03474aaa6724817eda7ec6 ({username='oryx', principal='oryx/127.21.151.60@KRBTEST.COM'} at 127.0.0.1:39368)
I20260619 20:27:37.671355 28253 tablet_copy_service.cc:161] P 9b72a6e0b4b14623beafb8fbeb362cdc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b187e099fc03474aaa6724817eda7ec6 at {username='oryx', principal='oryx/127.21.151.60@KRBTEST.COM'} at 127.0.0.1:39368: session id = b187e099fc03474aaa6724817eda7ec6-00000000000000000000000000000000
I20260619 20:27:37.671912 28253 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 20:27:37.672336 28889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 20:27:37.673453 28889 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: tablet copy: Starting download of 0 data blocks...
I20260619 20:27:37.673596 28889 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: tablet copy: Starting download of 1 WAL segments...
I20260619 20:27:37.674280 28889 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 20:27:37.675122 28253 tablet_copy_service.cc:342] P 9b72a6e0b4b14623beafb8fbeb362cdc: Request end of tablet copy session b187e099fc03474aaa6724817eda7ec6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.151.60@KRBTEST.COM'} at 127.0.0.1:39368
I20260619 20:27:37.675165 28253 tablet_copy_service.cc:433] P 9b72a6e0b4b14623beafb8fbeb362cdc: ending tablet copy session b187e099fc03474aaa6724817eda7ec6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b187e099fc03474aaa6724817eda7ec6
I20260619 20:27:37.677691 28889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:37.678771 28995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.678840 28998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:37.679363 28889 server_base.cc:1034] running on GCE node
W20260619 20:27:37.683693 28996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:37.683888 28889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:37.684031 28889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:37.685154 28889 hybrid_clock.cc:648] HybridClock initialized: now 1781900857685142 us; error 29 us; skew 500 ppm
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:37.686625 28889 init.cc:377] Logged in from keytab as oryx/127.21.151.60@KRBTEST.COM (short username oryx)
I20260619 20:27:37.686872 28889 webserver.cc:460] Webserver started at http://127.21.151.60:37949/ using document root <none> and password file <none>
I20260619 20:27:37.687000 28889 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:37.687036 28889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:37.688031 28889 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:27:37.688639 29005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.688768 28889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:37.688812 28889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal
uuid: "b187e099fc03474aaa6724817eda7ec6"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "897774d1d871d21a4e010194ac55605a"
server_key_iv: "0f38edd0deb2fc6af21cfea603afe85e"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.688875 28889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:37.703183 28889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:37.705158 28889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:37.708436 28889 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.60:35461
I20260619 20:27:37.708765 28889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
I20260619 20:27:37.711763 29058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.60:35461 every 8 connection(s)
I20260619 20:27:37.712608 29059 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:27:37.713088 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: Bootstrap starting.
I20260619 20:27:37.716027 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:27:37.716230 29059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: Bootstrap complete.
I20260619 20:27:37.716418 29059 raft_consensus.cc:348] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } }
I20260619 20:27:37.716509 29059 raft_consensus.cc:729] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b187e099fc03474aaa6724817eda7ec6, State: Initialized, Role: LEARNER
I20260619 20:27:37.716543 29059 consensus_queue.cc:260] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } }
I20260619 20:27:37.716722 29059 sys_catalog.cc:564] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:37.716724 29062 sys_catalog.cc:455] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:37.716974 29062 sys_catalog.cc:458] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: This master's current role is: LEARNER
I20260619 20:27:37.718269 29073 catalog_manager.cc:1259] Loaded cluster ID: 6bedd11f191a42d7b3296864b84fd211
I20260619 20:27:37.718555 29073 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: loading cluster ID for follower catalog manager: success
I20260619 20:27:37.719897 29073 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: acquiring CA information for follower catalog manager: success
I20260619 20:27:37.720224 29073 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 20:27:37.741514 28983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:37.741706 28983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:37.741731 28983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:37.742894 28983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:37.742939 28983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:37.742967 28983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:37.744594 28983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab
--principal=oryx/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:36487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=36845
--tserver_master_addrs=127.21.151.62:38657,127.21.151.61:37133,127.21.151.60:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:37.744829 28983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:37.745100 28983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:37.750012 29081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.750042 29079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:37.750234 28983 server_base.cc:1034] running on GCE node
W20260619 20:27:37.750003 29078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:37.750545 28983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:37.750787 28983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:37.751950 28983 hybrid_clock.cc:648] HybridClock initialized: now 1781900857751935 us; error 31 us; skew 500 ppm
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:37.754441 28983 init.cc:377] Logged in from keytab as oryx/127.21.151.1@KRBTEST.COM (short username oryx)
I20260619 20:27:37.754834 28983 webserver.cc:460] Webserver started at http://127.21.151.1:36845/ using document root <none> and password file <none>
I20260619 20:27:37.755051 28983 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:37.755132 28983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:37.756435 28983 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:37.756973 29088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.757133 28983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:37.757233 28983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "ad5b341bb7a44916b1943af42a35fd28"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "69c15973d256bd2bae2f3d4134c2af05"
server_key_iv: "c173e73678e36e282eca1bbaa0d67415"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.757572 28983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:37.766086 28983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:37.768231 28983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:37.768374 28983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:37.768579 28983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:37.768859 28983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:37.768908 28983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.768944 28983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:37.768968 28983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.774281 28983 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:36487
I20260619 20:27:37.774344 29201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:36487 every 8 connection(s)
I20260619 20:27:37.774669 28983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.60@KRBTEST.COM
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.61@KRBTEST.COM
I20260619 20:27:37.784788 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 28983
I20260619 20:27:37.784952 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 28430
I20260619 20:27:37.785995 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.775326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:38361 (local address 127.21.151.62:38657)
0619 20:27:37.775420 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.775422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.775829 (+ 407us) server_negotiation.cc:407] Connection header received
0619 20:27:37.776730 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.776731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.776745 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.776798 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.777584 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.778402 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.779555 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.779889 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.779958 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.782758 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.782759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.782760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.782767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.784380 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.784850 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.784850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.784851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.784988 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.785361 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.785362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.785362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.785498 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.785544 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.785825 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.785905 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 20:27:37.786274 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.775720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:35547 (local address 127.21.151.61:37133)
0619 20:27:37.775763 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.775765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.775862 (+ 97us) server_negotiation.cc:407] Connection header received
0619 20:27:37.776727 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.776728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.776741 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.776775 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.777624 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.778384 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.779757 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.780002 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.780090 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.783316 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.783318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.783318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.783327 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.784999 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.785421 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.785422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.785423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.785476 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.785668 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.785669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.785669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.785810 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.785849 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.786060 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.786141 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:37.786303 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.775343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.1:38361)
0619 20:27:37.775770 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.775780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.776606 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.776866 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.776870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.777265 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.777486 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.777490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.778508 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.778509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.779448 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.779450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.779971 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.779972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.780113 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.780713 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.780717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.782677 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.784454 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.784455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.784455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.784758 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.785261 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.785261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.785262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.785305 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.785587 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.785592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.785699 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.785805 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.785963 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":172,"thread_start_us":76,"threads_started":1}
I20260619 20:27:37.786720 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.775532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:59091 (local address 127.21.151.60:35461)
0619 20:27:37.775662 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.775665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.775677 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:27:37.776727 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.776728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.776741 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.776779 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.777592 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.778456 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.779727 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.780031 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.780121 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.782405 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.782406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.782407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.782415 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.784257 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.784849 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.784850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.784850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.784921 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.786156 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.786157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.786157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.786285 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.786350 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.786533 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.786625 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260619 20:27:37.786731 29202 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38657
I20260619 20:27:37.786808 29202 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:37.786972 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.775343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:35461 (local address 127.21.151.1:59091)
0619 20:27:37.775543 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.775582 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.776605 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.776866 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.776870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.777299 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.777487 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.777492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.778545 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.778545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.779650 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.779652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.780119 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.780120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.780182 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.780862 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.780867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.782309 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.784344 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.784350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.784352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.784758 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.785086 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.785087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.785087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.785213 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.786378 (+ 1165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.786379 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.786420 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.786790 (+ 370us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.786885 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":34,"thread_start_us":85,"threads_started":1}
I20260619 20:27:37.786975 29202 heartbeater.cc:511] Master 127.21.151.62:38657 requested a full tablet report, sending...
I20260619 20:27:37.787417 28232 ts_manager.cc:194] Re-registered known tserver with Master: ad5b341bb7a44916b1943af42a35fd28 (127.21.151.1:36487)
I20260619 20:27:37.787799 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.775666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:37133 (local address 127.21.151.1:35547)
0619 20:27:37.775811 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.775820 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.776613 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.776866 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.776870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.777296 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.777487 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.777492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.778513 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.778514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.779650 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.779652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.780152 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.780153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.780223 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.780703 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.780713 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.783184 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.785133 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.785134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.785134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.785345 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.785545 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.785546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.785546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.785603 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.785890 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.785891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.785933 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.787351 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.787471 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":122,"thread_start_us":59,"threads_started":1}
I20260619 20:27:37.787969 28232 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:38361
I20260619 20:27:37.788916 29203 heartbeater.cc:347] Connected to a master server at 127.21.151.60:35461
I20260619 20:27:37.788969 29203 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:37.789034 29203 heartbeater.cc:511] Master 127.21.151.60:35461 requested a full tablet report, sending...
I20260619 20:27:37.789119 29204 heartbeater.cc:347] Connected to a master server at 127.21.151.61:37133
I20260619 20:27:37.789150 29204 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:37.789186 29204 heartbeater.cc:511] Master 127.21.151.61:37133 requested a full tablet report, sending...
I20260619 20:27:37.789387 29022 ts_manager.cc:194] Registered new tserver with Master: ad5b341bb7a44916b1943af42a35fd28 (127.21.151.1:36487)
I20260619 20:27:37.789824 28835 ts_manager.cc:194] Registered new tserver with Master: ad5b341bb7a44916b1943af42a35fd28 (127.21.151.1:36487)
I20260619 20:27:37.792197 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:33131
--local_ip_for_outbound_sockets=127.21.151.2
--tserver_master_addrs=127.21.151.62:38657
--webserver_port=34819
--webserver_interface=127.21.151.2
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab
--principal=oryx/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.151.62:38657,127.21.151.61:37133
--tserver_master_addrs=127.21.151.62:38657,127.21.151.61:37133,127.21.151.60:35461 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:37.902635 29210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:37.902956 29210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:37.903007 29210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:37.905428 29210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:37.905488 29210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:37.905534 29210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:37.908453 29210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab
--principal=oryx/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:33131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=34819
--tserver_master_addrs=127.21.151.62:38657,127.21.151.61:37133,127.21.151.60:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:37.908715 29210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:37.908989 29210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:37.914294 29215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.914429 29218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:37.914306 29216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:37.914955 29210 server_base.cc:1034] running on GCE node
I20260619 20:27:37.915210 29210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:37.915481 29210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:37.916646 29210 hybrid_clock.cc:648] HybridClock initialized: now 1781900857916632 us; error 27 us; skew 500 ppm
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:37.919806 29210 init.cc:377] Logged in from keytab as oryx/127.21.151.2@KRBTEST.COM (short username oryx)
I20260619 20:27:37.920192 29210 webserver.cc:460] Webserver started at http://127.21.151.2:34819/ using document root <none> and password file <none>
I20260619 20:27:37.920406 29210 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:37.920485 29210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:37.921903 29210 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:27:37.922616 29225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.922806 29210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:27:37.922909 29210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "e0721bde923c430dbeea0ad061fd32eb"
format_stamp: "Formatted at 2026-06-19 20:27:36 on dist-test-slave-f9rk"
server_key: "5feb5a57a9e5ad26d54daee995449ec8"
server_key_iv: "4ecf2085d4edca347ab399e2222feb31"
server_key_version: "encryptionkey@0"
I20260619 20:27:37.923283 29210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:37.936029 29210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:37.938017 29210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:37.938161 29210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:37.938370 29210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:37.938658 29210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:37.938706 29210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.938745 29210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:37.938781 29210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:37.944453 29210 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:33131
I20260619 20:27:37.944496 29339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:33131 every 8 connection(s)
I20260619 20:27:37.944813 29210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:27:37.947256 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 29210
I20260619 20:27:37.947417 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 28565
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.60@KRBTEST.COM
I20260619 20:27:37.954593 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:44885
--local_ip_for_outbound_sockets=127.21.151.3
--tserver_master_addrs=127.21.151.62:38657
--webserver_port=46493
--webserver_interface=127.21.151.3
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab
--principal=oryx/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.151.62:38657,127.21.151.61:37133
--tserver_master_addrs=127.21.151.62:38657,127.21.151.61:37133,127.21.151.60:35461 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:27:37 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900857, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.61@KRBTEST.COM
I20260619 20:27:37.957579 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.945881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:43795 (local address 127.21.151.60:35461)
0619 20:27:37.945924 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.945926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.945934 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:37.946817 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.946818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.946834 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.946863 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.947965 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.948655 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.949886 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.950255 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.950302 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.952916 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.952917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.952918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.952926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.954708 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.956724 (+ 2016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.956726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.956726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.956786 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.956988 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.956989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.956989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.957130 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.957193 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.957395 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.957480 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:27:37.957592 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.945521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:47103 (local address 127.21.151.62:38657)
0619 20:27:37.945896 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.945898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.945907 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:37.946817 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.946819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.946831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.946863 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.947810 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.948577 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.949669 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.949870 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.949968 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.952916 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.952917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.952918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.952926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.955418 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.955900 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.955901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.955902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.955969 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.956249 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.956250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.956250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.956387 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.956438 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.957101 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.957190 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348}
I20260619 20:27:37.958218 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.945501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.2:47103)
0619 20:27:37.945660 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.945699 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.946699 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.946933 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.946936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.947405 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.947712 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.947718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.948696 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.948697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.949571 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.949573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.949982 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.949983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.950111 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.950621 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.950631 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.952806 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.955498 (+ 2692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.955499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.955499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.955805 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.956045 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.956045 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.956045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.956176 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.956482 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.956487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.956646 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.957704 (+ 1058us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.957819 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":45,"thread_start_us":85,"threads_started":1}
I20260619 20:27:37.958228 29346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.945778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:35461 (local address 127.21.151.2:43795)
0619 20:27:37.945887 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.945896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.946706 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.946933 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.946936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.947820 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.947874 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.947876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.948750 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.948751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.949792 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.949794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.950376 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.950377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.950448 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.950814 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.950819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.952806 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.954784 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.954790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.954792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.956643 (+ 1851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.956844 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.956844 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.956844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.956932 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.957222 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.957222 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.957265 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.957362 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.957525 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":23,"thread_start_us":65,"threads_started":1}
I20260619 20:27:37.958981 29340 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38657
I20260619 20:27:37.959141 29340 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:37.959343 29341 heartbeater.cc:347] Connected to a master server at 127.21.151.60:35461
I20260619 20:27:37.959343 29340 heartbeater.cc:511] Master 127.21.151.62:38657 requested a full tablet report, sending...
I20260619 20:27:37.959403 29341 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:37.959453 29341 heartbeater.cc:511] Master 127.21.151.60:35461 requested a full tablet report, sending...
I20260619 20:27:37.959736 28232 ts_manager.cc:194] Re-registered known tserver with Master: e0721bde923c430dbeea0ad061fd32eb (127.21.151.2:33131)
I20260619 20:27:37.959796 29022 ts_manager.cc:194] Registered new tserver with Master: e0721bde923c430dbeea0ad061fd32eb (127.21.151.2:33131)
I20260619 20:27:37.960240 28232 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:47103
I20260619 20:27:37.960423 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.945502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:37133 (local address 127.21.151.2:53703)
0619 20:27:37.945666 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:37.945699 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:37.946698 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:37.946932 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:37.946935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:37.947415 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:37.947714 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.947719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.953043 (+ 5324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.953044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.953964 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:37.953967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.954517 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.954518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:37.954582 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.954934 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:37.954939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:37.956413 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:37.958464 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.958465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.958466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.958890 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.959130 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:37.959131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:37.959131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:37.959199 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:37.959524 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:37.959525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:37.959571 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 20:27:37.959703 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 20:27:37.959818 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260619 20:27:37.960798 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:37.945530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:53703 (local address 127.21.151.61:37133)
0619 20:27:37.945574 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:37.945576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:37.945787 (+ 211us) server_negotiation.cc:407] Connection header received
0619 20:27:37.946793 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:37.946795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:37.946807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:37.946867 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:37.952185 (+ 5318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.952949 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.954063 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:37.954370 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:37.954417 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:37.956500 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:37.956502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:37.956502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:37.956511 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:37.958358 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.958971 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.958973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.958973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.959040 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:37.959273 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:37.959274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:37.959275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:37.959419 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:37.959496 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:37.960558 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0619 20:27:37.960667 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:37.961089 29342 heartbeater.cc:347] Connected to a master server at 127.21.151.61:37133
I20260619 20:27:37.961145 29342 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:37.961225 29342 heartbeater.cc:511] Master 127.21.151.61:37133 requested a full tablet report, sending...
I20260619 20:27:37.961535 28835 ts_manager.cc:194] Registered new tserver with Master: e0721bde923c430dbeea0ad061fd32eb (127.21.151.2:33131)
W20260619 20:27:38.036737 29347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:38.036934 29347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:38.036963 29347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:38.038049 29347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:38.038084 29347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:38.038110 29347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:38.040270 29347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab
--principal=oryx/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:44885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=46493
--tserver_master_addrs=127.21.151.62:38657,127.21.151.61:37133,127.21.151.60:35461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:38.040519 29347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:38.040784 29347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:27:38.041558 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.038606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.21.151.60:35461)
0619 20:27:38.038668 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.038670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.038680 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:38.038731 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.038731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.038749 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.038804 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:38.038959 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.039464 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.040799 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.041225 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.041283 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.041328 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.041471 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.041499 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:27:38.041600 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.038542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:35461 (local address 127.0.0.1:57318)
0619 20:27:38.038595 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:38.038608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:38.038681 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:38.038832 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:38.038833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:38.038835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:38.038889 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.038891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.039544 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.039545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.040709 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.040713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.041313 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.041314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.041388 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.041393 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:27:38.041469 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:27:38.041519 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 20:27:38.041971 29033 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 20:27:38.042132 29062 sys_catalog.cc:455] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b72a6e0b4b14623beafb8fbeb362cdc. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:38.042251 29062 sys_catalog.cc:458] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: This master's current role is: LEARNER
W20260619 20:27:38.045966 29352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:38.045993 29353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:38.045995 29355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:38.046185 29347 server_base.cc:1034] running on GCE node
I20260619 20:27:38.046382 29347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:38.046571 29347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:38.047672 29347 hybrid_clock.cc:648] HybridClock initialized: now 1781900858047653 us; error 27 us; skew 500 ppm
Jun 19 20:27:38 dist-test-slave-f9rk krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900858, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:38.050138 29347 init.cc:377] Logged in from keytab as oryx/127.21.151.3@KRBTEST.COM (short username oryx)
I20260619 20:27:38.050571 29347 webserver.cc:460] Webserver started at http://127.21.151.3:46493/ using document root <none> and password file <none>
I20260619 20:27:38.050812 29347 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:38.050884 29347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:38.052261 29347 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:38.052836 29362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:38.053025 29347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:38.053119 29347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "d1cbe361770b455492864ccae615cf28"
format_stamp: "Formatted at 2026-06-19 20:27:37 on dist-test-slave-f9rk"
server_key: "6939c9e1197e3052fcc9292a2b2c25fe"
server_key_iv: "8654220d6bea2b959a15a407a4d3e713"
server_key_version: "encryptionkey@0"
I20260619 20:27:38.053474 29347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:38.069657 29347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:38.071421 29347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:38.071588 29347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:38.071856 29347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:38.072158 29347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:38.072211 29347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:38.072245 29347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:38.072273 29347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:38.077909 29347 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:44885
I20260619 20:27:38.077970 29475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:44885 every 8 connection(s)
I20260619 20:27:38.078362 29347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:38.079929 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 29347
Jun 19 20:27:38 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900858, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.61@KRBTEST.COM
Jun 19 20:27:38 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900858, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
Jun 19 20:27:38 dist-test-slave-f9rk krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900858, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.60@KRBTEST.COM
I20260619 20:27:38.091677 28886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.079377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:33583 (local address 127.21.151.61:37133)
0619 20:27:38.079416 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.079418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.079520 (+ 102us) server_negotiation.cc:407] Connection header received
0619 20:27:38.080032 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.080033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.080046 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.080080 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:38.081322 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.082082 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.083712 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.084022 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.084061 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.088328 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:38.088329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:38.088330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:38.088337 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:38.090000 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.090554 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.090555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.090555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.090612 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.090908 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.090909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.090909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.091038 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:38.091094 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.091447 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.091537 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:27:38.091698 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.079113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:54389 (local address 127.21.151.60:35461)
0619 20:27:38.079205 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.079207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.079331 (+ 124us) server_negotiation.cc:407] Connection header received
0619 20:27:38.080036 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.080038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.080056 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.080091 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:38.081406 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.082164 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.084274 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.084570 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.084612 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.088363 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:38.088365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:38.088365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:38.088371 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:38.090086 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.090553 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.090554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.090555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.090612 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.090903 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.090904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.090904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.091019 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:38.091065 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.091452 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.091548 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 20:27:38.092439 29480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.079026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.3:53737)
0619 20:27:38.079276 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:38.079286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:38.079910 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:38.080272 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:38.080277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:38.080930 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:38.081213 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.081218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.082205 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.082206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.084112 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.084114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.084674 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.084675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.084739 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.085325 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:38.085329 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:38.088064 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:38.090674 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:38.090674 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:38.090675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:38.091226 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:38.091434 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:38.091435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:38.091435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:38.091728 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:38.092053 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:38.092053 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:38.092090 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 20:27:38.092140 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:27:38.092251 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":965,"thread_start_us":94,"threads_started":1}
I20260619 20:27:38.092521 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.079155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:53737 (local address 127.21.151.62:38657)
0619 20:27:38.079246 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.079248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.079339 (+ 91us) server_negotiation.cc:407] Connection header received
0619 20:27:38.080043 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.080045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.080059 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.080089 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:38.081323 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.082089 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.084269 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.084570 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.084625 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.088142 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:38.088143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:38.088144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:38.088151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:38.090594 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.091304 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.091305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.091305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.091365 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.091810 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.091812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.091812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.091975 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:38.092029 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.092346 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.092430 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":67}
I20260619 20:27:38.092258 29481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.079028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:35461 (local address 127.21.151.3:54389)
0619 20:27:38.079213 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:38.079255 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:38.079914 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:38.080274 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:38.080278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:38.081239 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:38.081309 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.081311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.082267 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.082268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.084174 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.084177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.084681 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.084682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.084737 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.085274 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:38.085279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:38.088281 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:38.090161 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:38.090162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:38.090162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:38.090463 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:38.090679 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:38.090679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:38.090680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:38.090828 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:38.091105 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:38.091110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:38.091287 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 20:27:38.091819 (+ 532us) client_negotiation.cc:240] Negotiation successful
0619 20:27:38.092008 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":948,"thread_start_us":90,"threads_started":1}
I20260619 20:27:38.093163 29477 heartbeater.cc:347] Connected to a master server at 127.21.151.60:35461
I20260619 20:27:38.093174 29476 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38657
I20260619 20:27:38.093312 29476 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:38.093464 29482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.079328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:37133 (local address 127.21.151.3:33583)
0619 20:27:38.079440 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:38.079448 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:38.079919 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:38.080272 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:38.080277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:38.080930 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:38.081213 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.081218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.082205 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.082206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.083567 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.083569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.084285 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.084286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.084442 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.084988 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:38.084998 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:38.088225 (+ 3227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:38.090089 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:38.090095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:38.090097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:38.090465 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:38.090675 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:38.090675 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:38.090675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:38.090828 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:38.091163 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:38.091164 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:38.091289 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 20:27:38.092148 (+ 859us) client_negotiation.cc:240] Negotiation successful
0619 20:27:38.092307 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":123,"thread_start_us":60,"threads_started":1}
I20260619 20:27:38.093487 29476 heartbeater.cc:511] Master 127.21.151.62:38657 requested a full tablet report, sending...
I20260619 20:27:38.093312 29477 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:38.093904 29477 heartbeater.cc:511] Master 127.21.151.60:35461 requested a full tablet report, sending...
I20260619 20:27:38.093940 29478 heartbeater.cc:347] Connected to a master server at 127.21.151.61:37133
I20260619 20:27:38.093983 29478 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:38.094038 29478 heartbeater.cc:511] Master 127.21.151.61:37133 requested a full tablet report, sending...
I20260619 20:27:38.094055 28232 ts_manager.cc:194] Re-registered known tserver with Master: d1cbe361770b455492864ccae615cf28 (127.21.151.3:44885)
I20260619 20:27:38.094178 29022 ts_manager.cc:194] Registered new tserver with Master: d1cbe361770b455492864ccae615cf28 (127.21.151.3:44885)
I20260619 20:27:38.094466 28232 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:53737
I20260619 20:27:38.094516 28835 ts_manager.cc:194] Registered new tserver with Master: d1cbe361770b455492864ccae615cf28 (127.21.151.3:44885)
I20260619 20:27:38.094964 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.081279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.21.151.62:38657)
0619 20:27:38.081559 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.081560 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.081712 (+ 152us) server_negotiation.cc:407] Connection header received
0619 20:27:38.081733 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.081734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.081745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.081798 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:38.082512 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.083280 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.084303 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.085189 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.085524 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.086689 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:38.086690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:38.086691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:38.086697 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:38.093120 (+ 6423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.093576 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.093577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.093577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.093633 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.093848 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.093849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.093850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.093981 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:38.094162 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.094761 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.094841 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":255,"thread_start_us":79,"threads_started":1}
I20260619 20:27:38.097992 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.082858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.21.151.60:35461)
0619 20:27:38.083065 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.083067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.083154 (+ 87us) server_negotiation.cc:407] Connection header received
0619 20:27:38.083235 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.083236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.083245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.083280 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:38.086588 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.087350 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.090231 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.090527 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.090561 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.095209 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:38.095211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:38.095211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:38.095218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:38.096839 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.097166 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.097167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.097167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.097217 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.097424 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.097425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.097425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.097531 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:38.097625 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.097797 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.097876 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":48,"threads_started":1}
I20260619 20:27:38.098424 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.082838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.21.151.61:37133)
0619 20:27:38.082926 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.082928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.083136 (+ 208us) server_negotiation.cc:407] Connection header received
0619 20:27:38.083221 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.083222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.083232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.083267 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:38.088202 (+ 4935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.088883 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.090871 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.091158 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.091212 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.095499 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:38.095501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:38.095501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:38.095508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:38.097127 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.097477 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.097477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.097478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.097528 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.097728 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.097729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.097730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.097859 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:38.098244 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.098286 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.098356 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":45,"threads_started":1}
I20260619 20:27:38.096760 28232 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:38.099182 28232 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:38.099926 28987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: NON_VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: true } } }
I20260619 20:27:38.100039 28987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:38.101269 28987 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc: attempting to promote NON_VOTER b187e099fc03474aaa6724817eda7ec6 to VOTER
I20260619 20:27:38.101384 28987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } }
I20260619 20:27:38.101770 28845 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Refusing update from remote peer 9b72a6e0b4b14623beafb8fbeb362cdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 20:27:38.101950 29033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 LEARNER]: Refusing update from remote peer 9b72a6e0b4b14623beafb8fbeb362cdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 20:27:38.102137 28796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 20:27:38.102207 28795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 20:27:38.102835 28987 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b187e099fc03474aaa6724817eda7ec6 (127.21.151.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:38.103025 28845 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b187e099fc03474aaa6724817eda7ec6 (127.21.151.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:38.103039 29033 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b187e099fc03474aaa6724817eda7ec6 (127.21.151.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:38.103718 28986 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:38.103727 28987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:38.103783 28986 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:27:38.103803 28987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:38.103866 28987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:38.103915 28987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:38.103950 28987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:38.103991 28987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:38.103992 29062 sys_catalog.cc:455] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:38.104061 29062 sys_catalog.cc:458] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:27:38.108646 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.104831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:33131 (local address 127.0.0.1:42492)
0619 20:27:38.104877 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:38.104886 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:38.104920 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:38.105569 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:38.105570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:38.105571 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:38.105603 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.105604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.106586 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.106587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.107652 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.107654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.108444 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.108444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.108503 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.108507 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:38.108541 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 20:27:38.108583 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:27:38.108675 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.104936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.21.151.2:33131)
0619 20:27:38.105358 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.105368 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.105375 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:27:38.105429 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.105431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.105446 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.105568 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:38.105653 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.106507 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.107721 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.108343 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.108440 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.108522 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.108562 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.108599 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":77,"threads_started":1}
I20260619 20:27:38.108964 29500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.105019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:44885 (local address 127.0.0.1:34958)
0619 20:27:38.105174 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:38.105182 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:38.105232 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:38.105582 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:38.105583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:38.105584 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:38.105616 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.105617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.106879 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.106880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.107757 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.107759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.108690 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.108691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.108803 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.108806 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:38.108861 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:27:38.108894 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260619 20:27:38.108999 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.105083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.21.151.3:44885)
0619 20:27:38.105194 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.105198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.105224 (+ 26us) server_negotiation.cc:407] Connection header received
0619 20:27:38.105315 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.105317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.105332 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.105433 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:38.105684 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.106785 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.107835 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.108540 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.108597 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.108660 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.108889 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.108930 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260619 20:27:38.109349 29274 tablet_service.cc:1467] Processing CreateTablet for tablet 075a3ee381f24dd7950a2e691ab22aac (DEFAULT_TABLE table=test-table [id=3db751de4c7c49fd8c5f65298d8f4649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:38.109640 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.104831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:36487 (local address 127.0.0.1:46006)
0619 20:27:38.105122 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:38.105132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:38.105217 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:38.105423 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:38.105423 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:38.105425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:38.105487 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.105489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.106864 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.106865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.108015 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.108017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.109406 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.109407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.109480 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.109484 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:38.109534 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:27:38.109575 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":48,"thread_start_us":81,"threads_started":1}
I20260619 20:27:38.109601 29410 tablet_service.cc:1467] Processing CreateTablet for tablet 075a3ee381f24dd7950a2e691ab22aac (DEFAULT_TABLE table=test-table [id=3db751de4c7c49fd8c5f65298d8f4649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:38.109705 29274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075a3ee381f24dd7950a2e691ab22aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:38.109856 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.104936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.21.151.1:36487)
0619 20:27:38.105125 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.105130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.105190 (+ 60us) server_negotiation.cc:407] Connection header received
0619 20:27:38.105325 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.105328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.105342 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.105424 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:38.105715 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.106787 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.108093 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.109281 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.109402 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.109474 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.109589 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.109628 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260619 20:27:38.109869 29410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075a3ee381f24dd7950a2e691ab22aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:38.113194 29505 tablet_bootstrap.cc:492] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: Bootstrap starting.
I20260619 20:27:38.113749 29505 tablet_bootstrap.cc:654] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:38.113767 29136 tablet_service.cc:1467] Processing CreateTablet for tablet 075a3ee381f24dd7950a2e691ab22aac (DEFAULT_TABLE table=test-table [id=3db751de4c7c49fd8c5f65298d8f4649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:38.114100 29505 log.cc:824] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:38.114114 29136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 075a3ee381f24dd7950a2e691ab22aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:38.114431 29506 tablet_bootstrap.cc:492] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: Bootstrap starting.
I20260619 20:27:38.115132 29506 tablet_bootstrap.cc:654] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:38.115428 29506 log.cc:824] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:38.116231 29505 tablet_bootstrap.cc:492] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: No bootstrap required, opened a new log
I20260619 20:27:38.116302 29505 ts_tablet_manager.cc:1397] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:27:38.116629 29506 tablet_bootstrap.cc:492] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: No bootstrap required, opened a new log
I20260619 20:27:38.116755 29506 ts_tablet_manager.cc:1397] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:27:38.119962 29509 tablet_bootstrap.cc:492] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: Bootstrap starting.
I20260619 20:27:38.120378 29506 raft_consensus.cc:348] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:38.120561 29506 raft_consensus.cc:374] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:38.120613 29506 raft_consensus.cc:729] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0721bde923c430dbeea0ad061fd32eb, State: Initialized, Role: FOLLOWER
I20260619 20:27:38.120631 29509 tablet_bootstrap.cc:654] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:38.120728 29506 consensus_queue.cc:260] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:38.120923 29509 log.cc:824] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:38.120980 29506 ts_tablet_manager.cc:1428] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260619 20:27:38.121064 29340 heartbeater.cc:503] Master 127.21.151.62:38657 was elected leader, sending a full tablet report...
I20260619 20:27:38.121145 29505 raft_consensus.cc:348] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:38.121411 29505 raft_consensus.cc:374] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:38.121493 29505 raft_consensus.cc:729] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1cbe361770b455492864ccae615cf28, State: Initialized, Role: FOLLOWER
I20260619 20:27:38.121621 29505 consensus_queue.cc:260] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:38.121871 29505 ts_tablet_manager.cc:1428] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260619 20:27:38.121992 29476 heartbeater.cc:503] Master 127.21.151.62:38657 was elected leader, sending a full tablet report...
I20260619 20:27:38.123800 29509 tablet_bootstrap.cc:492] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: No bootstrap required, opened a new log
I20260619 20:27:38.123867 29509 ts_tablet_manager.cc:1397] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 20:27:38.126492 29509 raft_consensus.cc:348] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:38.126663 29509 raft_consensus.cc:374] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:38.126705 29509 raft_consensus.cc:729] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad5b341bb7a44916b1943af42a35fd28, State: Initialized, Role: FOLLOWER
I20260619 20:27:38.126816 29509 consensus_queue.cc:260] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:38.127094 29202 heartbeater.cc:503] Master 127.21.151.62:38657 was elected leader, sending a full tablet report...
I20260619 20:27:38.127122 29509 ts_tablet_manager.cc:1428] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260619 20:27:38.195545 29343 tablet.cc:2369] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:27:38.275673 29205 tablet.cc:2369] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:27:38.301270 29513 raft_consensus.cc:482] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:38.301378 29513 raft_consensus.cc:504] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:38.301800 29513 leader_election.cc:288] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0721bde923c430dbeea0ad061fd32eb (127.21.151.2:33131), d1cbe361770b455492864ccae615cf28 (127.21.151.3:44885)
I20260619 20:27:38.305722 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.302038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:50455 (local address 127.21.151.3:44885)
0619 20:27:38.302091 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.302093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.302171 (+ 78us) server_negotiation.cc:407] Connection header received
0619 20:27:38.302312 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.302313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.302325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.302394 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:38.302609 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.303327 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.304789 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.305370 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.305436 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.305444 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.305614 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.305651 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:27:38.305722 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.301998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:34003 (local address 127.21.151.2:33131)
0619 20:27:38.302047 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.302050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.302133 (+ 83us) server_negotiation.cc:407] Connection header received
0619 20:27:38.302259 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.302260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.302268 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.302334 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:38.302489 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.303259 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.304789 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.305343 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.305436 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.305444 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.305614 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.305657 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:27:38.305751 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.301983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:44885 (local address 127.21.151.1:50455)
0619 20:27:38.302093 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:38.302103 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:38.302166 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:38.302446 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:38.302447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:38.302449 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:38.302533 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.302535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.303393 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.303393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.304682 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.304685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.305451 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.305452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.305533 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.305535 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:27:38.305595 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 20:27:38.305641 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":61,"thread_start_us":53,"threads_started":1}
I20260619 20:27:38.305758 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.301950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:33131 (local address 127.21.151.1:34003)
0619 20:27:38.302073 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:38.302083 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:38.302181 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:38.302338 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:38.302339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:38.302342 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:38.302412 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.302414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.303343 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.303344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.304682 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:38.304685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.305449 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.305451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:38.305530 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.305535 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:27:38.305595 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 20:27:38.305641 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":46,"thread_start_us":59,"threads_started":1}
I20260619 20:27:38.306075 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075a3ee381f24dd7950a2e691ab22aac" candidate_uuid: "ad5b341bb7a44916b1943af42a35fd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0721bde923c430dbeea0ad061fd32eb" is_pre_election: true
I20260619 20:27:38.306262 29294 raft_consensus.cc:2454] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad5b341bb7a44916b1943af42a35fd28 in term 0.
I20260619 20:27:38.306391 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075a3ee381f24dd7950a2e691ab22aac" candidate_uuid: "ad5b341bb7a44916b1943af42a35fd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1cbe361770b455492864ccae615cf28" is_pre_election: true
I20260619 20:27:38.306486 29092 leader_election.cc:302] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad5b341bb7a44916b1943af42a35fd28, e0721bde923c430dbeea0ad061fd32eb; no voters:
I20260619 20:27:38.306527 29430 raft_consensus.cc:2454] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad5b341bb7a44916b1943af42a35fd28 in term 0.
I20260619 20:27:38.306627 29513 raft_consensus.cc:2781] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:38.306680 29513 raft_consensus.cc:482] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:38.306701 29513 raft_consensus.cc:3037] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:38.307367 29513 raft_consensus.cc:504] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:38.307564 29513 leader_election.cc:288] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [CANDIDATE]: Term 1 election: Requested vote from peers e0721bde923c430dbeea0ad061fd32eb (127.21.151.2:33131), d1cbe361770b455492864ccae615cf28 (127.21.151.3:44885)
I20260619 20:27:38.307652 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075a3ee381f24dd7950a2e691ab22aac" candidate_uuid: "ad5b341bb7a44916b1943af42a35fd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1cbe361770b455492864ccae615cf28"
I20260619 20:27:38.307740 29430 raft_consensus.cc:3037] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:38.307699 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "075a3ee381f24dd7950a2e691ab22aac" candidate_uuid: "ad5b341bb7a44916b1943af42a35fd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0721bde923c430dbeea0ad061fd32eb"
I20260619 20:27:38.307835 29294 raft_consensus.cc:3037] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:38.308449 29430 raft_consensus.cc:2454] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad5b341bb7a44916b1943af42a35fd28 in term 1.
I20260619 20:27:38.308590 29294 raft_consensus.cc:2454] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad5b341bb7a44916b1943af42a35fd28 in term 1.
I20260619 20:27:38.308676 29089 leader_election.cc:302] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad5b341bb7a44916b1943af42a35fd28, d1cbe361770b455492864ccae615cf28; no voters:
I20260619 20:27:38.308761 29513 raft_consensus.cc:2781] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:38.308905 29513 raft_consensus.cc:686] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 1 LEADER]: Becoming Leader. State: Replica: ad5b341bb7a44916b1943af42a35fd28, State: Running, Role: LEADER
I20260619 20:27:38.309098 29513 consensus_queue.cc:237] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:38.309952 28232 catalog_manager.cc:5495] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad5b341bb7a44916b1943af42a35fd28 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "ad5b341bb7a44916b1943af42a35fd28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } health_report { overall_health: UNKNOWN } } }
W20260619 20:27:38.329147 29479 tablet.cc:2369] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:27:38.337224 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.334815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46016 (local address 127.21.151.1:36487)
0619 20:27:38.334883 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.334884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.334894 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:38.334974 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.334975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.334997 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.335066 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:38.335207 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.335749 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.336481 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.336670 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.336740 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.336919 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:38.337017 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:38.337101 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.337123 (+ 22us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.337144 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:27:38.338876 29430 raft_consensus.cc:1261] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28 [term 1 FOLLOWER]: Refusing update from remote peer ad5b341bb7a44916b1943af42a35fd28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:38.338900 29294 raft_consensus.cc:1261] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb [term 1 FOLLOWER]: Refusing update from remote peer ad5b341bb7a44916b1943af42a35fd28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:38.339234 29516 consensus_queue.cc:1035] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:38.339365 29513 consensus_queue.cc:1035] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:38.344523 28232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:38.344619 28232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:38.345830 28232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b72a6e0b4b14623beafb8fbeb362cdc: Sending DeleteTablet for 3 replicas of tablet 075a3ee381f24dd7950a2e691ab22aac
I20260619 20:27:38.346180 29410 tablet_service.cc:1514] Processing DeleteTablet for tablet 075a3ee381f24dd7950a2e691ab22aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:38 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:34958
I20260619 20:27:38.346203 29274 tablet_service.cc:1514] Processing DeleteTablet for tablet 075a3ee381f24dd7950a2e691ab22aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:38 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:42492
I20260619 20:27:38.346406 29530 tablet_replica.cc:330] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: stopping tablet replica
I20260619 20:27:38.346395 29136 tablet_service.cc:1514] Processing DeleteTablet for tablet 075a3ee381f24dd7950a2e691ab22aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:38 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:46006
I20260619 20:27:38.346482 29530 raft_consensus.cc:2229] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:38.346586 29530 raft_consensus.cc:2258] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:38.346596 29531 tablet_replica.cc:330] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: stopping tablet replica
I20260619 20:27:38.346679 29531 raft_consensus.cc:2229] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:38.346742 29529 tablet_replica.cc:330] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: stopping tablet replica
I20260619 20:27:38.346760 29531 raft_consensus.cc:2258] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:38.346822 29529 raft_consensus.cc:2229] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:38.346906 29529 raft_consensus.cc:2258] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:38.347067 29530 ts_tablet_manager.cc:1905] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:38.347335 29529 ts_tablet_manager.cc:1905] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:38.348305 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 28200
I20260619 20:27:38.348912 29530 ts_tablet_manager.cc:1918] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:38.348985 29530 log.cc:1196] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/075a3ee381f24dd7950a2e691ab22aac
I20260619 20:27:38.349241 29530 ts_tablet_manager.cc:1939] T 075a3ee381f24dd7950a2e691ab22aac P e0721bde923c430dbeea0ad061fd32eb: Deleting consensus metadata
I20260619 20:27:38.349874 29529 ts_tablet_manager.cc:1918] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:38.349932 29529 log.cc:1196] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/075a3ee381f24dd7950a2e691ab22aac
I20260619 20:27:38.350189 29529 ts_tablet_manager.cc:1939] T 075a3ee381f24dd7950a2e691ab22aac P d1cbe361770b455492864ccae615cf28: Deleting consensus metadata
I20260619 20:27:38.354916 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.347801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.21.151.60:35461)
0619 20:27:38.347865 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.347867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.347876 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:38.347895 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.347896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.347908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.347960 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:38.348793 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.349548 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.350642 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.350855 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.350940 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.352323 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:38.352325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:38.352325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:38.352333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:38.353787 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.354140 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.354141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.354141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.354203 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.354409 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.354410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.354410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.354523 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:38.354624 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.354728 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.354813 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 20:27:38.355748 29364 connection.cc:535] server connection from 127.0.0.1:34958 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:27:38.355907 29531 ts_tablet_manager.cc:1905] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:27:38.355989 29228 connection.cc:535] server connection from 127.0.0.1:42492 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:27:38.357482 29531 ts_tablet_manager.cc:1918] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:38.357560 29531 log.cc:1196] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/075a3ee381f24dd7950a2e691ab22aac
I20260619 20:27:38.357797 29531 ts_tablet_manager.cc:1939] T 075a3ee381f24dd7950a2e691ab22aac P ad5b341bb7a44916b1943af42a35fd28: Deleting consensus metadata
W20260619 20:27:38.358156 29089 connection.cc:441] server connection from 127.0.0.1:46006 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46006 (request call id 1) could send its response
I20260619 20:27:38.365527 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.358304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.21.151.61:37133)
0619 20:27:38.358346 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.358348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.358448 (+ 100us) server_negotiation.cc:407] Connection header received
0619 20:27:38.358673 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.358674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.358688 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.358740 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:38.359304 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.359949 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.361090 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.361282 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.361362 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.362621 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:38.362622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:38.362623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:38.362630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:38.364289 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.364690 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.364691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.364691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.364753 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.365001 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.365002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.365003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.365119 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:38.365220 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.365322 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.365401 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:27:38.366389 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:38.358548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.21.151.60:35461)
0619 20:27:38.358583 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:38.358584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:38.358660 (+ 76us) server_negotiation.cc:407] Connection header received
0619 20:27:38.358747 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:38.358748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:38.358757 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:38.358798 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:38.359787 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.360326 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.362043 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:38.362332 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:38.362383 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:38.363715 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:38.363716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:38.363717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:38.363724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:38.365334 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.365706 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.365707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.365707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.365756 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:38.365963 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:38.365964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:38.365964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:38.366071 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:38.366141 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:38.366276 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 20:27:38.366330 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:39.343674 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.343409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.2:46307)
0619 20:27:39.343537 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.343632 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
W20260619 20:27:39.343967 29340 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:38657 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
I20260619 20:27:39.344269 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.344029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.1:54657)
0619 20:27:39.344163 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.344227 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":53,"threads_started":1}
W20260619 20:27:39.344461 29202 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:38657 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
I20260619 20:27:39.348213 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.348001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.3:40897)
0619 20:27:39.348105 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.348169 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
W20260619 20:27:39.348404 29476 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:38657 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
I20260619 20:27:39.846306 29542 raft_consensus.cc:482] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b72a6e0b4b14623beafb8fbeb362cdc)
I20260619 20:27:39.846427 29542 raft_consensus.cc:504] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } }
I20260619 20:27:39.846745 29542 leader_election.cc:288] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657), a34fef0e27f1436fb3af99ad4c4baeaf (127.21.151.61:37133)
I20260619 20:27:39.846791 29543 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9b72a6e0b4b14623beafb8fbeb362cdc)
I20260619 20:27:39.846913 29543 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } }
I20260619 20:27:39.847116 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.846851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39852)
0619 20:27:39.847025 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.847076 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":48,"threads_started":1}
I20260619 20:27:39.847169 29543 leader_election.cc:288] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657), b187e099fc03474aaa6724817eda7ec6 (127.21.151.60:35461)
W20260619 20:27:39.847493 29009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
I20260619 20:27:39.847545 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.847249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39858)
0619 20:27:39.847456 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.847497 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":56,"threads_started":1}
W20260619 20:27:39.847658 28822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
I20260619 20:27:39.847946 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.847861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39872)
0619 20:27:39.847897 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.847920 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 20:27:39.848043 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.847958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39878)
0619 20:27:39.847989 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.848022 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260619 20:27:39.848102 28822 leader_election.cc:334] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657): Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
W20260619 20:27:39.848150 29009 leader_election.cc:334] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657): Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
I20260619 20:27:39.850082 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.846893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:37133 (local address 127.0.0.1:60308)
0619 20:27:39.847136 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.847144 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:39.847223 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:39.847461 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:39.847463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:39.847465 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:39.847546 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:39.847548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:39.848524 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:39.848525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:39.849282 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:39.849284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:39.849847 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:39.849848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:39.849910 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:39.849915 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:27:39.849970 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:27:39.850020 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":148,"threads_started":1}
I20260619 20:27:39.850085 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.846944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.21.151.61:37133)
0619 20:27:39.847054 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:39.847057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:39.847281 (+ 224us) server_negotiation.cc:407] Connection header received
0619 20:27:39.847309 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:39.847310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:39.847321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:39.847408 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:39.847642 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:39.848451 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:39.849376 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:39.849750 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:39.849844 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:39.849898 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:39.849980 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 20:27:39.850016 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":47,"threads_started":1}
I20260619 20:27:39.850318 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.847289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:35461 (local address 127.0.0.1:57356)
0619 20:27:39.847803 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.847814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:39.847862 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:39.848070 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:39.848071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:39.848073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:39.848121 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:39.848124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:39.848768 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:39.848769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:39.849498 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:39.849500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:39.850042 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:39.850043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:39.850101 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:39.850105 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:39.850170 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 20:27:39.850225 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":38,"threads_started":1}
I20260619 20:27:39.850343 28845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b187e099fc03474aaa6724817eda7ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" is_pre_election: true
I20260619 20:27:39.850402 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.847670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.21.151.60:35461)
0619 20:27:39.847943 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:39.847944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:39.847951 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:27:39.847971 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:39.847971 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:39.847980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:39.848180 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:39.848209 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:39.848684 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:39.849587 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:39.849948 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:39.850039 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:39.850049 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:39.850289 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 20:27:39.850333 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":72,"threads_started":1}
I20260619 20:27:39.850443 28845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b187e099fc03474aaa6724817eda7ec6 in term 1.
I20260619 20:27:39.850579 29033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b187e099fc03474aaa6724817eda7ec6" is_pre_election: true
I20260619 20:27:39.850617 29007 leader_election.cc:302] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a34fef0e27f1436fb3af99ad4c4baeaf, b187e099fc03474aaa6724817eda7ec6; no voters: 9b72a6e0b4b14623beafb8fbeb362cdc
I20260619 20:27:39.850675 29033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34fef0e27f1436fb3af99ad4c4baeaf in term 1.
I20260619 20:27:39.850732 29542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 20:27:39.850775 29542 raft_consensus.cc:482] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9b72a6e0b4b14623beafb8fbeb362cdc)
I20260619 20:27:39.850800 29542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:27:39.850831 28820 leader_election.cc:302] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a34fef0e27f1436fb3af99ad4c4baeaf, b187e099fc03474aaa6724817eda7ec6; no voters: 9b72a6e0b4b14623beafb8fbeb362cdc
I20260619 20:27:39.850921 29543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 20:27:39.850952 29543 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9b72a6e0b4b14623beafb8fbeb362cdc)
I20260619 20:27:39.850971 29543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:27:39.858649 29542 raft_consensus.cc:504] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } }
I20260619 20:27:39.858671 29543 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } }
I20260619 20:27:39.858801 29542 leader_election.cc:288] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657), a34fef0e27f1436fb3af99ad4c4baeaf (127.21.151.61:37133)
I20260619 20:27:39.858820 29543 leader_election.cc:288] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 2 election: Requested vote from peers 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657), b187e099fc03474aaa6724817eda7ec6 (127.21.151.60:35461)
I20260619 20:27:39.859030 28845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b187e099fc03474aaa6724817eda7ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf"
I20260619 20:27:39.859107 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.858962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39884)
0619 20:27:39.859001 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.859025 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 20:27:39.859148 28845 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b187e099fc03474aaa6724817eda7ec6 in current term 2: Already voted for candidate a34fef0e27f1436fb3af99ad4c4baeaf in this term.
I20260619 20:27:39.859292 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.859180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39900)
0619 20:27:39.859231 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.859261 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:27:39.859300 29033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b187e099fc03474aaa6724817eda7ec6"
I20260619 20:27:39.859403 29033 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a34fef0e27f1436fb3af99ad4c4baeaf in current term 2: Already voted for candidate b187e099fc03474aaa6724817eda7ec6 in this term.
I20260619 20:27:39.859790 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.859703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39916)
0619 20:27:39.859749 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.859773 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:27:39.859884 28822 leader_election.cc:334] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657): Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
I20260619 20:27:39.859930 28822 leader_election.cc:302] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a34fef0e27f1436fb3af99ad4c4baeaf; no voters: 9b72a6e0b4b14623beafb8fbeb362cdc, b187e099fc03474aaa6724817eda7ec6
I20260619 20:27:39.860020 29543 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 20:27:39.860072 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:39.859968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:39924)
0619 20:27:39.860025 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:39.860048 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 20:27:39.860126 29009 leader_election.cc:334] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657): Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
I20260619 20:27:39.860163 29009 leader_election.cc:302] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b187e099fc03474aaa6724817eda7ec6; no voters: 9b72a6e0b4b14623beafb8fbeb362cdc, a34fef0e27f1436fb3af99ad4c4baeaf
I20260619 20:27:39.860247 29542 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 20:27:40.344535 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:40.344281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.2:50653)
0619 20:27:40.344432 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:40.344479 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260619 20:27:40.345001 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:40.344800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.1:47229)
0619 20:27:40.344915 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:40.344955 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20260619 20:27:40.348858 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:40.348703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.3:59831)
0619 20:27:40.348792 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:40.348824 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":43,"threads_started":1}
I20260619 20:27:41.345266 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.345010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.2:58383)
0619 20:27:41.345153 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.345192 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260619 20:27:41.345554 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.345370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.1:48329)
0619 20:27:41.345484 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.345515 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":51,"threads_started":1}
I20260619 20:27:41.349385 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.349228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.3:42987)
0619 20:27:41.349322 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.349352 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":46,"threads_started":1}
I20260619 20:27:41.648193 29558 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:41.648278 29558 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } }
I20260619 20:27:41.648502 29558 leader_election.cc:288] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657), b187e099fc03474aaa6724817eda7ec6 (127.21.151.60:35461)
I20260619 20:27:41.648734 29033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b187e099fc03474aaa6724817eda7ec6" is_pre_election: true
I20260619 20:27:41.648813 29033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34fef0e27f1436fb3af99ad4c4baeaf in term 2.
I20260619 20:27:41.648824 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.648626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:36594)
0619 20:27:41.648738 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.648769 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":60,"threads_started":1}
I20260619 20:27:41.648949 28820 leader_election.cc:302] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a34fef0e27f1436fb3af99ad4c4baeaf, b187e099fc03474aaa6724817eda7ec6; no voters:
I20260619 20:27:41.649046 29558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 20:27:41.649085 29558 raft_consensus.cc:482] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:41.649118 29558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 2 FOLLOWER]: Advancing to term 3
I20260619 20:27:41.649232 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.649139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:36606)
0619 20:27:41.649177 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.649206 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 20:27:41.649312 28822 leader_election.cc:334] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657): Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
I20260619 20:27:41.649868 29558 raft_consensus.cc:504] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } }
I20260619 20:27:41.649989 29558 leader_election.cc:288] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 3 election: Requested vote from peers 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657), b187e099fc03474aaa6724817eda7ec6 (127.21.151.60:35461)
I20260619 20:27:41.650135 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.650050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:36620)
0619 20:27:41.650085 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.650113 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 20:27:41.650146 29033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b187e099fc03474aaa6724817eda7ec6"
I20260619 20:27:41.650240 29033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260619 20:27:41.650421 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.650335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:36624)
0619 20:27:41.650378 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.650401 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:27:41.650496 28822 leader_election.cc:334] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657): Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
I20260619 20:27:41.650804 29033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34fef0e27f1436fb3af99ad4c4baeaf in term 3.
I20260619 20:27:41.650945 28820 leader_election.cc:302] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a34fef0e27f1436fb3af99ad4c4baeaf, b187e099fc03474aaa6724817eda7ec6; no voters: 9b72a6e0b4b14623beafb8fbeb362cdc
I20260619 20:27:41.651031 29558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 3 FOLLOWER]: Leader election won for term 3
I20260619 20:27:41.651124 29558 raft_consensus.cc:686] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [term 3 LEADER]: Becoming Leader. State: Replica: a34fef0e27f1436fb3af99ad4c4baeaf, State: Running, Role: LEADER
I20260619 20:27:41.651216 29558 consensus_queue.cc:237] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } }
I20260619 20:27:41.651575 29561 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34fef0e27f1436fb3af99ad4c4baeaf. Latest consensus state: current_term: 3 leader_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:41.651695 29561 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:41.651870 29563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:41.652287 29563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:41.652374 29563 catalog_manager.cc:1259] Loaded cluster ID: 6bedd11f191a42d7b3296864b84fd211
I20260619 20:27:41.652405 29563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:41.652582 29563 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:41.652671 29563 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Loaded TSK: 0
I20260619 20:27:41.652808 29563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:27:41.684369 28834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:41.685911 28834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:41.686977 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.686855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:36638)
0619 20:27:41.686909 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.686935 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:27:41.686978 29033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [term 3 FOLLOWER]: Refusing update from remote peer a34fef0e27f1436fb3af99ad4c4baeaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 20:27:41.687222 29561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 20:27:41.687345 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.687247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.0.0.1:36650)
0619 20:27:41.687288 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.687311 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 20:27:41.687426 28822 consensus_peers.cc:487] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf -> Peer 9b72a6e0b4b14623beafb8fbeb362cdc (127.21.151.62:38657): Couldn't send request to peer 9b72a6e0b4b14623beafb8fbeb362cdc. Status: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 20:27:41.687868 29565 sys_catalog.cc:455] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a34fef0e27f1436fb3af99ad4c4baeaf. Latest consensus state: current_term: 3 leader_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:41.687975 29565 sys_catalog.cc:458] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:27:41.688364 29564 mvcc.cc:205] Tried to move back new op lower bound from 7298665929467985920 to 7298665929324032000. Current Snapshot: MvccSnapshot[applied={T|T < 7298665929467985920}]
I20260619 20:27:41.688517 29558 sys_catalog.cc:455] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:41.688536 29565 sys_catalog.cc:455] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9b72a6e0b4b14623beafb8fbeb362cdc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38657 } } peers { permanent_uuid: "a34fef0e27f1436fb3af99ad4c4baeaf" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 37133 } attrs { promote: false } } peers { permanent_uuid: "b187e099fc03474aaa6724817eda7ec6" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 35461 } attrs { promote: false } } }
I20260619 20:27:41.688608 29558 sys_catalog.cc:458] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:41.688616 29565 sys_catalog.cc:458] T 00000000000000000000000000000000 P b187e099fc03474aaa6724817eda7ec6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:27:41.688695 29566 mvcc.cc:205] Tried to move back new op lower bound from 7298665929467985920 to 7298665929324032000. Current Snapshot: MvccSnapshot[applied={T|T < 7298665929467985920 or (T in {7298665929467985920})}]
I20260619 20:27:41.693459 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.690466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:44885 (local address 127.0.0.1:53172)
0619 20:27:41.690551 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.690560 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:41.690598 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:41.690751 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:41.690752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:41.690754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:41.690827 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:41.690829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.691735 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.691735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:41.692605 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:41.692608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.693230 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.693230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:41.693296 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:41.693300 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:41.693353 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:27:41.693390 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":112,"thread_start_us":58,"threads_started":1}
I20260619 20:27:41.693470 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.690528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.21.151.3:44885)
0619 20:27:41.690639 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:41.690641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:41.690649 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:41.690672 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:41.690673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:41.690683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:41.690827 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:41.690872 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.691656 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.692698 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.693150 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.693219 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:41.693232 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:41.693368 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 20:27:41.693403 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260619 20:27:41.693948 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.690233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:33131 (local address 127.0.0.1:57448)
0619 20:27:41.690272 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.690284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:41.690329 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:41.690702 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:41.690704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:41.690705 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:41.690753 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:41.690754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.691738 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.691738 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:41.692939 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:41.692941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.693734 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.693735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:41.693800 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:41.693804 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:41.693853 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:27:41.693892 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":82}
I20260619 20:27:41.693953 29410 tablet_service.cc:1467] Processing CreateTablet for tablet ee467f6c9d8e4f4999fe452f111c68bd (DEFAULT_TABLE table=test-table [id=d8196a0df3c94f85ac748e56cd0da0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:41.694024 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.690296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.21.151.2:33131)
0619 20:27:41.690582 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:41.690584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:41.690593 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:41.690617 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:41.690618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:41.690628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:41.690688 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:41.690852 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.691643 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.693026 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.693640 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.693731 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:41.693743 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:41.693877 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 20:27:41.693933 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":60,"threads_started":1}
I20260619 20:27:41.694083 29410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee467f6c9d8e4f4999fe452f111c68bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:41.694169 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.690275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:36487 (local address 127.0.0.1:60430)
0619 20:27:41.690397 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.690405 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:41.690441 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:41.690681 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:41.690682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:41.690685 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:41.690740 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:41.690743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.691733 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.691734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:41.693170 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:41.693171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.693959 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.693960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:41.694019 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:41.694023 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:41.694054 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 20:27:41.694100 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":34,"thread_start_us":72,"threads_started":1}
I20260619 20:27:41.694414 29572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.690418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.21.151.1:36487)
0619 20:27:41.690518 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:41.690520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:41.690530 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:41.690558 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:41.690559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:41.690571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:41.690932 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:41.690986 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.691484 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.693267 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:41.693871 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:41.694277 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:41.694294 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:41.694314 (+ 20us) server_negotiation.cc:299] Negotiation successful
0619 20:27:41.694365 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
I20260619 20:27:41.695389 29575 tablet_bootstrap.cc:492] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28: Bootstrap starting.
I20260619 20:27:41.695570 29274 tablet_service.cc:1467] Processing CreateTablet for tablet ee467f6c9d8e4f4999fe452f111c68bd (DEFAULT_TABLE table=test-table [id=d8196a0df3c94f85ac748e56cd0da0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:41.695713 29274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee467f6c9d8e4f4999fe452f111c68bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:41.695732 29136 tablet_service.cc:1467] Processing CreateTablet for tablet ee467f6c9d8e4f4999fe452f111c68bd (DEFAULT_TABLE table=test-table [id=d8196a0df3c94f85ac748e56cd0da0c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:41.695837 29136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee467f6c9d8e4f4999fe452f111c68bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:41.695863 29575 tablet_bootstrap.cc:654] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:41.696667 29577 tablet_bootstrap.cc:492] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb: Bootstrap starting.
I20260619 20:27:41.697038 29577 tablet_bootstrap.cc:654] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:41.697216 29578 tablet_bootstrap.cc:492] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28: Bootstrap starting.
I20260619 20:27:41.697516 29575 tablet_bootstrap.cc:492] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28: No bootstrap required, opened a new log
I20260619 20:27:41.697568 29575 ts_tablet_manager.cc:1397] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:41.697623 29577 tablet_bootstrap.cc:492] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb: No bootstrap required, opened a new log
I20260619 20:27:41.697655 29578 tablet_bootstrap.cc:654] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:41.697669 29577 ts_tablet_manager.cc:1397] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:41.697688 29575 raft_consensus.cc:348] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:41.697754 29575 raft_consensus.cc:374] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:41.697773 29575 raft_consensus.cc:729] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1cbe361770b455492864ccae615cf28, State: Initialized, Role: FOLLOWER
I20260619 20:27:41.697775 29577 raft_consensus.cc:348] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:41.697837 29577 raft_consensus.cc:374] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:41.697862 29577 raft_consensus.cc:729] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0721bde923c430dbeea0ad061fd32eb, State: Initialized, Role: FOLLOWER
I20260619 20:27:41.697846 29575 consensus_queue.cc:260] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:41.697908 29577 consensus_queue.cc:260] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:41.698036 29575 ts_tablet_manager.cc:1428] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:41.698114 29577 ts_tablet_manager.cc:1428] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:41.698385 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.698292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.2:49409)
0619 20:27:41.698326 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.698358 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 20:27:41.698427 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.698323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.3:43927)
0619 20:27:41.698363 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.698394 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 20:27:41.698729 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.698624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.2:58835)
0619 20:27:41.698676 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.698705 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 20:27:41.698714 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.698638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.3:48607)
0619 20:27:41.698672 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.698692 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 20:27:41.699753 29478 heartbeater.cc:503] Master 127.21.151.61:37133 was elected leader, sending a full tablet report...
I20260619 20:27:41.699887 29342 heartbeater.cc:503] Master 127.21.151.61:37133 was elected leader, sending a full tablet report...
I20260619 20:27:41.700173 29578 tablet_bootstrap.cc:492] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28: No bootstrap required, opened a new log
I20260619 20:27:41.700227 29578 ts_tablet_manager.cc:1397] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:27:41.700330 29578 raft_consensus.cc:348] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:41.700413 29578 raft_consensus.cc:374] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:41.700444 29578 raft_consensus.cc:729] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad5b341bb7a44916b1943af42a35fd28, State: Initialized, Role: FOLLOWER
I20260619 20:27:41.700496 29578 consensus_queue.cc:260] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:41.700698 29578 ts_tablet_manager.cc:1428] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:41.700907 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.700822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.1:33787)
0619 20:27:41.700854 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.700878 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260619 20:27:41.701138 29204 heartbeater.cc:503] Master 127.21.151.61:37133 was elected leader, sending a full tablet report...
I20260619 20:27:41.701193 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:41.701110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.1:57425)
0619 20:27:41.701140 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:41.701167 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260619 20:27:42.033116 29583 raft_consensus.cc:482] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:42.033213 29583 raft_consensus.cc:504] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:42.033370 29583 leader_election.cc:288] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0721bde923c430dbeea0ad061fd32eb (127.21.151.2:33131), d1cbe361770b455492864ccae615cf28 (127.21.151.3:44885)
I20260619 20:27:42.033640 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee467f6c9d8e4f4999fe452f111c68bd" candidate_uuid: "ad5b341bb7a44916b1943af42a35fd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0721bde923c430dbeea0ad061fd32eb" is_pre_election: true
I20260619 20:27:42.033650 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee467f6c9d8e4f4999fe452f111c68bd" candidate_uuid: "ad5b341bb7a44916b1943af42a35fd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1cbe361770b455492864ccae615cf28" is_pre_election: true
I20260619 20:27:42.033746 29294 raft_consensus.cc:2454] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad5b341bb7a44916b1943af42a35fd28 in term 0.
I20260619 20:27:42.033742 29430 raft_consensus.cc:2454] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad5b341bb7a44916b1943af42a35fd28 in term 0.
I20260619 20:27:42.033927 29089 leader_election.cc:302] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad5b341bb7a44916b1943af42a35fd28, d1cbe361770b455492864ccae615cf28; no voters:
I20260619 20:27:42.034078 29583 raft_consensus.cc:2781] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:42.034132 29583 raft_consensus.cc:482] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:42.034147 29583 raft_consensus.cc:3037] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:42.034845 29583 raft_consensus.cc:504] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:42.034981 29583 leader_election.cc:288] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [CANDIDATE]: Term 1 election: Requested vote from peers e0721bde923c430dbeea0ad061fd32eb (127.21.151.2:33131), d1cbe361770b455492864ccae615cf28 (127.21.151.3:44885)
I20260619 20:27:42.035145 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee467f6c9d8e4f4999fe452f111c68bd" candidate_uuid: "ad5b341bb7a44916b1943af42a35fd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1cbe361770b455492864ccae615cf28"
I20260619 20:27:42.035142 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee467f6c9d8e4f4999fe452f111c68bd" candidate_uuid: "ad5b341bb7a44916b1943af42a35fd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0721bde923c430dbeea0ad061fd32eb"
I20260619 20:27:42.035238 29430 raft_consensus.cc:3037] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:42.035239 29294 raft_consensus.cc:3037] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:42.036034 29294 raft_consensus.cc:2454] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad5b341bb7a44916b1943af42a35fd28 in term 1.
I20260619 20:27:42.036033 29430 raft_consensus.cc:2454] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad5b341bb7a44916b1943af42a35fd28 in term 1.
I20260619 20:27:42.036224 29092 leader_election.cc:302] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad5b341bb7a44916b1943af42a35fd28, e0721bde923c430dbeea0ad061fd32eb; no voters:
I20260619 20:27:42.036321 29583 raft_consensus.cc:2781] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:42.036470 29583 raft_consensus.cc:686] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 1 LEADER]: Becoming Leader. State: Replica: ad5b341bb7a44916b1943af42a35fd28, State: Running, Role: LEADER
I20260619 20:27:42.036514 29583 consensus_queue.cc:237] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } } peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } }
I20260619 20:27:42.036895 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:42.036706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.1:39695)
0619 20:27:42.036792 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:42.036834 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 20:27:42.037246 28834 catalog_manager.cc:5495] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad5b341bb7a44916b1943af42a35fd28 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "ad5b341bb7a44916b1943af42a35fd28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad5b341bb7a44916b1943af42a35fd28" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 36487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 } health_report { overall_health: UNKNOWN } } }
I20260619 20:27:42.047194 29572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:42.044977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.21.151.1:36487)
0619 20:27:42.045037 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:42.045040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:42.045050 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:42.045097 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:42.045098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:42.045116 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:42.045175 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:42.045313 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:42.045775 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:42.046516 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:42.046703 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:42.046778 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:42.046922 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:42.046949 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:42.047006 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:42.047091 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:27:42.047111 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:27:42.048063 29430 raft_consensus.cc:1261] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28 [term 1 FOLLOWER]: Refusing update from remote peer ad5b341bb7a44916b1943af42a35fd28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:42.048075 29294 raft_consensus.cc:1261] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb [term 1 FOLLOWER]: Refusing update from remote peer ad5b341bb7a44916b1943af42a35fd28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:42.048266 29583 consensus_queue.cc:1035] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1cbe361770b455492864ccae615cf28" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 44885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:42.048430 29584 consensus_queue.cc:1035] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0721bde923c430dbeea0ad061fd32eb" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 33131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:42.049016 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:42.048897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.3:52253)
0619 20:27:42.048938 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:42.048984 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 20:27:42.049300 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:42.049180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.2:38841)
0619 20:27:42.049232 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:42.049263 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 20:27:42.049955 29589 mvcc.cc:205] Tried to move back new op lower bound from 7298665930947227648 to 7298665930902286336. Current Snapshot: MvccSnapshot[applied={T|T < 7298665930947227648}]
I20260619 20:27:42.050256 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:42.050172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.3:59219)
0619 20:27:42.050208 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:42.050232 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 20:27:42.050884 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:42.050764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.1:50683)
0619 20:27:42.050823 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:42.050852 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:27:42.051287 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:42.051200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.1:39867)
0619 20:27:42.051234 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:42.051258 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 20:27:42.051326 29590 mvcc.cc:205] Tried to move back new op lower bound from 7298665930947227648 to 7298665930902286336. Current Snapshot: MvccSnapshot[applied={T|T < 7298665930947227648 or (T in {7298665930947227648})}]
I20260619 20:27:42.051705 29592 mvcc.cc:205] Tried to move back new op lower bound from 7298665930947227648 to 7298665930902286336. Current Snapshot: MvccSnapshot[applied={T|T < 7298665930947227648}]
I20260619 20:27:42.051753 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:42.051455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38657 (local address 127.21.151.2:43621)
0619 20:27:42.051582 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:42.051731 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 20:27:42.051962 28835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:42.052049 28835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:42.053238 28835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a34fef0e27f1436fb3af99ad4c4baeaf: Sending DeleteTablet for 3 replicas of tablet ee467f6c9d8e4f4999fe452f111c68bd
I20260619 20:27:42.053543 29274 tablet_service.cc:1514] Processing DeleteTablet for tablet ee467f6c9d8e4f4999fe452f111c68bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:42 UTC) from {username='oryx', principal='oryx/127.21.151.61@KRBTEST.COM'} at 127.0.0.1:57448
I20260619 20:27:42.053557 29410 tablet_service.cc:1514] Processing DeleteTablet for tablet ee467f6c9d8e4f4999fe452f111c68bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:42 UTC) from {username='oryx', principal='oryx/127.21.151.61@KRBTEST.COM'} at 127.0.0.1:53172
I20260619 20:27:42.053604 29136 tablet_service.cc:1514] Processing DeleteTablet for tablet ee467f6c9d8e4f4999fe452f111c68bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:42 UTC) from {username='oryx', principal='oryx/127.21.151.61@KRBTEST.COM'} at 127.0.0.1:60430
I20260619 20:27:42.053767 29598 tablet_replica.cc:330] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb: stopping tablet replica
I20260619 20:27:42.053831 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 28983
I20260619 20:27:42.053936 29600 tablet_replica.cc:330] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28: stopping tablet replica
I20260619 20:27:42.053860 29598 raft_consensus.cc:2229] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:42.053989 29600 raft_consensus.cc:2229] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:42.053998 29598 raft_consensus.cc:2258] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:42.054042 29600 raft_consensus.cc:2258] T ee467f6c9d8e4f4999fe452f111c68bd P ad5b341bb7a44916b1943af42a35fd28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:42.054126 29599 tablet_replica.cc:330] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28: stopping tablet replica
I20260619 20:27:42.054167 29599 raft_consensus.cc:2229] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:42.054204 29599 raft_consensus.cc:2258] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:42.054713 29599 ts_tablet_manager.cc:1905] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:42.055408 29598 ts_tablet_manager.cc:1905] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:42.056254 29599 ts_tablet_manager.cc:1918] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:42.056318 29599 log.cc:1196] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/ee467f6c9d8e4f4999fe452f111c68bd
I20260619 20:27:42.056564 29599 ts_tablet_manager.cc:1939] T ee467f6c9d8e4f4999fe452f111c68bd P d1cbe361770b455492864ccae615cf28: Deleting consensus metadata
I20260619 20:27:42.056849 28819 catalog_manager.cc:4841] TS d1cbe361770b455492864ccae615cf28 (127.21.151.3:44885): tablet ee467f6c9d8e4f4999fe452f111c68bd (table test-table [id=d8196a0df3c94f85ac748e56cd0da0c6]) successfully deleted
I20260619 20:27:42.056998 29598 ts_tablet_manager.cc:1918] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:42.057045 29598 log.cc:1196] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/ee467f6c9d8e4f4999fe452f111c68bd
I20260619 20:27:42.057310 29598 ts_tablet_manager.cc:1939] T ee467f6c9d8e4f4999fe452f111c68bd P e0721bde923c430dbeea0ad061fd32eb: Deleting consensus metadata
I20260619 20:27:42.057649 28822 catalog_manager.cc:4841] TS e0721bde923c430dbeea0ad061fd32eb (127.21.151.2:33131): tablet ee467f6c9d8e4f4999fe452f111c68bd (table test-table [id=d8196a0df3c94f85ac748e56cd0da0c6]) successfully deleted
W20260619 20:27:42.061545 29570 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:42.061444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:36487 (local address 127.0.0.1:60446)
0619 20:27:42.061477 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:42.061503 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:36487: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":17}
W20260619 20:27:42.061715 28819 catalog_manager.cc:4584] TS ad5b341bb7a44916b1943af42a35fd28 (127.21.151.1:36487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee467f6c9d8e4f4999fe452f111c68bd: Network error: Client connection negotiation failed: client connection to 127.21.151.1:36487: connect: Connection reset by peer (error 104)
I20260619 20:27:42.061801 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 29210
I20260619 20:27:42.066980 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 29347
I20260619 20:27:42.071715 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 28699
I20260619 20:27:42.076767 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 28889
2026-06-19T20:27:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6731 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:42 dist-test-slave-f9rk krb5kdc[29603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:42 dist-test-slave-f9rk krb5kdc[29603](info): set up 2 sockets
Jun 19 20:27:42 dist-test-slave-f9rk krb5kdc[29603](info): commencing operation
krb5kdc: starting...
W20260619 20:27:43.112762 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.62.keytab.
Entry for principal oryx/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.62.keytab.
I20260619 20:27:43.222481 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44561
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.62.keytab
--principal=oryx/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:44561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:43.301090 29616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:43.301331 29616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:43.301367 29616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:43.302523 29616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:43.302594 29616 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 20:27:43.302615 29616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:43.302636 29616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:43.302670 29616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:43.302695 29616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:43.304246 29616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:44561
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.62.keytab
--principal=oryx/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:43.304528 29616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:43.304754 29616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:43.309357 29621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:43.309388 29624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:43.309398 29622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:43.309568 29616 server_base.cc:1034] running on GCE node
I20260619 20:27:43.310011 29616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:43.310283 29616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:43.311451 29616 hybrid_clock.cc:648] HybridClock initialized: now 1781900863311434 us; error 40 us; skew 500 ppm
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:43.313905 29616 init.cc:377] Logged in from keytab as oryx/127.21.151.62@KRBTEST.COM (short username oryx)
I20260619 20:27:43.314312 29616 webserver.cc:460] Webserver started at http://127.21.151.62:38417/ using document root <none> and password file <none>
I20260619 20:27:43.314545 29616 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:43.314615 29616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:43.314764 29616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:43.315871 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "7dafe6cf95014fa09b1d34d89d2594e6"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "fe9b6edd16414eb83a33948ccf5ba5db"
server_key_iv: "b12fe663b1659ce1cb282438f47f79b1"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.316213 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "7dafe6cf95014fa09b1d34d89d2594e6"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "fe9b6edd16414eb83a33948ccf5ba5db"
server_key_iv: "b12fe663b1659ce1cb282438f47f79b1"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.317508 29616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 20:27:43.318876 29631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:43.319073 29616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:43.319170 29616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "7dafe6cf95014fa09b1d34d89d2594e6"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "fe9b6edd16414eb83a33948ccf5ba5db"
server_key_iv: "b12fe663b1659ce1cb282438f47f79b1"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.319257 29616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:43.330119 29616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:43.332501 29616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:43.332661 29616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:43.337005 29616 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:44561
I20260619 20:27:43.337111 29694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:44561 every 8 connection(s)
I20260619 20:27:43.337407 29616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:43.338114 29695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:43.340354 29695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6: Bootstrap starting.
I20260619 20:27:43.340965 29695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:43.341257 29695 log.cc:824] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:43.341893 29695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6: No bootstrap required, opened a new log
I20260619 20:27:43.343928 29695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dafe6cf95014fa09b1d34d89d2594e6" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44561 } }
I20260619 20:27:43.344060 29695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:43.344081 29695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dafe6cf95014fa09b1d34d89d2594e6, State: Initialized, Role: FOLLOWER
I20260619 20:27:43.344245 29695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dafe6cf95014fa09b1d34d89d2594e6" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44561 } }
I20260619 20:27:43.344323 29695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:43.344384 29695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:43.344431 29695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:43.344945 29695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dafe6cf95014fa09b1d34d89d2594e6" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44561 } }
I20260619 20:27:43.345053 29695 leader_election.cc:302] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dafe6cf95014fa09b1d34d89d2594e6; no voters:
I20260619 20:27:43.345232 29695 leader_election.cc:288] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:43.345274 29698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:43.345418 29698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [term 1 LEADER]: Becoming Leader. State: Replica: 7dafe6cf95014fa09b1d34d89d2594e6, State: Running, Role: LEADER
I20260619 20:27:43.345482 29695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:43.345505 29698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dafe6cf95014fa09b1d34d89d2594e6" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44561 } }
I20260619 20:27:43.345912 29700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dafe6cf95014fa09b1d34d89d2594e6. Latest consensus state: current_term: 1 leader_uuid: "7dafe6cf95014fa09b1d34d89d2594e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dafe6cf95014fa09b1d34d89d2594e6" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44561 } } }
I20260619 20:27:43.345919 29699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dafe6cf95014fa09b1d34d89d2594e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dafe6cf95014fa09b1d34d89d2594e6" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44561 } } }
I20260619 20:27:43.346122 29699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:43.346103 29700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:43.346421 29707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:43.346822 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 29616
I20260619 20:27:43.346912 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:43.346961 29707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:43.347134 22108 external_mini_cluster.cc:1428] Setting key d4b144f73c6b64921019bea6e5718ff1
I20260619 20:27:43.348556 29707 catalog_manager.cc:1347] Generated new cluster ID: af59aff536834decb97b5e9e63992fe3
I20260619 20:27:43.348605 29707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:43.359166 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.347568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.21.151.62:44561)
0619 20:27:43.347793 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.347802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.347879 (+ 77us) server_negotiation.cc:407] Connection header received
0619 20:27:43.348053 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.348060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.348120 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.348276 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:43.349031 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.350384 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.351506 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.352678 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.352782 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.355508 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:43.355512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:43.355512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:43.355525 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:43.357464 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.357831 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.357834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.357834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.357936 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.358133 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.358134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.358134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.358434 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:43.358653 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.358795 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.358934 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":114,"threads_started":1}
I20260619 20:27:43.373173 29707 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:43.373903 29707 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:43.380839 29707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6: Generated new TSK 0
I20260619 20:27:43.381019 29707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab.
Entry for principal oryx/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab.
I20260619 20:27:43.410050 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44561
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab
--principal=oryx/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:43.490185 29721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:43.490417 29721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:43.490453 29721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:43.491667 29721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:43.491736 29721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:43.491770 29721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:27:43.491811 29721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:43.493281 29721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab
--principal=oryx/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.151.62:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:43.493543 29721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:43.493767 29721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:43.498683 29726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:43.498677 29729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:43.498677 29727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:43.498847 29721 server_base.cc:1034] running on GCE node
I20260619 20:27:43.499202 29721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:43.499450 29721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:43.500625 29721 hybrid_clock.cc:648] HybridClock initialized: now 1781900863500608 us; error 50 us; skew 500 ppm
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:43.503000 29721 init.cc:377] Logged in from keytab as oryx/127.21.151.1@KRBTEST.COM (short username oryx)
I20260619 20:27:43.503404 29721 webserver.cc:460] Webserver started at http://127.21.151.1:46639/ using document root <none> and password file <none>
I20260619 20:27:43.503690 29721 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:43.503758 29721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:43.503913 29721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:43.504937 29721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "76ec625d85604a7aa35181ce9c479292"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "dd8e4b256c09ff4bc065ae6f82c76a4c"
server_key_iv: "d08069fcb3ddc6f194ea53c81b3718cc"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.505287 29721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "76ec625d85604a7aa35181ce9c479292"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "dd8e4b256c09ff4bc065ae6f82c76a4c"
server_key_iv: "d08069fcb3ddc6f194ea53c81b3718cc"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.506605 29721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:27:43.507295 29736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:43.507467 29721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:27:43.507561 29721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "76ec625d85604a7aa35181ce9c479292"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "dd8e4b256c09ff4bc065ae6f82c76a4c"
server_key_iv: "d08069fcb3ddc6f194ea53c81b3718cc"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.507676 29721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:43.517012 29721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:43.518633 29721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:43.518779 29721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:43.519225 29721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:43.519286 29721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:43.519325 29721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:43.519351 29721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:43.526263 29721 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:37119
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:43.526332 29852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:37119 every 8 connection(s)
I20260619 20:27:43.526620 29721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:27:43.529513 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.519701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:48913 (local address 127.21.151.62:44561)
0619 20:27:43.519775 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.519777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.519971 (+ 194us) server_negotiation.cc:407] Connection header received
0619 20:27:43.520491 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.520493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.520505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.520594 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:43.521254 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.522132 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.523826 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.524179 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.524251 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.526787 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:43.526789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:43.526789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:43.526799 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:43.528109 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.528524 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.528525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.528526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.528586 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.528846 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.528847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.528847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.528965 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:43.529036 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.529311 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.529394 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:27:43.529655 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.519687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44561 (local address 127.21.151.1:48913)
0619 20:27:43.519854 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:43.519895 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:43.520368 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:43.520668 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:43.520672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:43.520972 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:43.521146 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.521150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.522243 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.522245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.523727 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.523731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.524270 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.524271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.524411 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.524885 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:43.524895 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:43.526705 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:43.528208 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.528215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.528217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.528432 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.528662 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.528663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.528663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.528772 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.529038 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:43.529045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:43.529176 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 20:27:43.529279 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 20:27:43.529432 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260619 20:27:43.530182 29853 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44561
I20260619 20:27:43.530261 29853 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:43.530452 29853 heartbeater.cc:511] Master 127.21.151.62:44561 requested a full tablet report, sending...
I20260619 20:27:43.530973 29647 ts_manager.cc:194] Registered new tserver with Master: 76ec625d85604a7aa35181ce9c479292 (127.21.151.1:37119)
I20260619 20:27:43.531710 29647 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:48913
I20260619 20:27:43.534739 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 29721
I20260619 20:27:43.534852 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:43.535079 22108 external_mini_cluster.cc:1428] Setting key f7a4610f4623d561ea4f8445a8ed4066
I20260619 20:27:43.536602 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.531162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44561 (local address 127.21.151.1:59451)
0619 20:27:43.531309 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:43.531325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:43.531392 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:43.531513 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:43.531514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:43.531759 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:43.531827 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.531830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.532497 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.532498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.532960 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.532961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.533255 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.533256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.533295 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.533630 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:43.533634 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:43.533883 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:43.535608 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.535609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.535610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.535844 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.536070 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.536070 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.536070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.536123 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.536383 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:43.536384 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:43.536418 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 20:27:43.536467 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:27:43.536541 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260619 20:27:43.536705 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.531212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:59451 (local address 127.21.151.62:44561)
0619 20:27:43.531256 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.531258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.531379 (+ 121us) server_negotiation.cc:407] Connection header received
0619 20:27:43.531455 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.531456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.531465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.531512 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:43.531905 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.532433 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.533033 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.533203 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.533256 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.533954 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:43.533955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:43.533955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:43.533960 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:43.535536 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.535930 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.535931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.535932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.536005 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.536188 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.536189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.536189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.536323 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:43.536389 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.536490 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.536573 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for oryx/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab.
Entry for principal oryx/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab.
I20260619 20:27:43.560418 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44561
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab
--principal=oryx/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:43.639519 29862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:43.639787 29862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:43.639825 29862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:43.641010 29862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:43.641080 29862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:43.641114 29862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:27:43.641155 29862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:43.642632 29862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab
--principal=oryx/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.151.62:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:43.642897 29862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:43.643122 29862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:43.648027 29870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:43.648036 29868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:43.648042 29867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:43.648304 29862 server_base.cc:1034] running on GCE node
I20260619 20:27:43.648507 29862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:43.648687 29862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:43.649817 29862 hybrid_clock.cc:648] HybridClock initialized: now 1781900863649805 us; error 28 us; skew 500 ppm
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:43.652181 29862 init.cc:377] Logged in from keytab as oryx/127.21.151.2@KRBTEST.COM (short username oryx)
I20260619 20:27:43.652594 29862 webserver.cc:460] Webserver started at http://127.21.151.2:43153/ using document root <none> and password file <none>
I20260619 20:27:43.652824 29862 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:43.652890 29862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:43.653043 29862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:43.654029 29862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "08126e498c2d4ae0b776ad4f138c2f37"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "2c81b7d375a54b3e3a472940ce84dc44"
server_key_iv: "c1e0477cf3d80f800c712b3337b877ea"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.654407 29862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "08126e498c2d4ae0b776ad4f138c2f37"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "2c81b7d375a54b3e3a472940ce84dc44"
server_key_iv: "c1e0477cf3d80f800c712b3337b877ea"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.655687 29862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:27:43.656384 29877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:43.656544 29862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:43.656630 29862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "08126e498c2d4ae0b776ad4f138c2f37"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "2c81b7d375a54b3e3a472940ce84dc44"
server_key_iv: "c1e0477cf3d80f800c712b3337b877ea"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.656705 29862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:43.666182 29862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:43.667876 29862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:43.668028 29862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:43.668524 29862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:43.668583 29862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:43.668614 29862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:43.668645 29862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:43.675288 29862 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:37603
I20260619 20:27:43.675362 29993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:37603 every 8 connection(s)
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:43.675715 29862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:27:43.678241 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.668983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:44989 (local address 127.21.151.62:44561)
0619 20:27:43.669065 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.669067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.669223 (+ 156us) server_negotiation.cc:407] Connection header received
0619 20:27:43.669674 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.669675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.669689 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.669773 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:43.670608 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.671455 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.673040 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.673365 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.673454 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.676127 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:43.676128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:43.676129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:43.676138 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:43.677213 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.677564 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.677565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.677565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.677611 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.677782 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.677783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.677783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.677874 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:43.677923 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.678127 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.678181 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 20:27:43.678484 29887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.668918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44561 (local address 127.21.151.2:44989)
0619 20:27:43.669075 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:43.669112 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:43.669577 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:43.669818 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:43.669821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:43.670204 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:43.670481 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.670488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.671564 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.671565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.672921 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.672943 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.673457 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.673457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.673574 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.674102 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:43.674111 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:43.676028 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:43.677278 (+ 1250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.677284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.677285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.677495 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.677654 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.677654 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.677655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.677733 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.677927 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:43.677932 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:43.678037 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 20:27:43.678115 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:27:43.678222 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260619 20:27:43.678956 29994 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44561
I20260619 20:27:43.679041 29994 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:43.679209 29994 heartbeater.cc:511] Master 127.21.151.62:44561 requested a full tablet report, sending...
I20260619 20:27:43.679610 29647 ts_manager.cc:194] Registered new tserver with Master: 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603)
I20260619 20:27:43.680128 29647 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:44989
I20260619 20:27:43.684563 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 29862
I20260619 20:27:43.684675 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:43.684859 22108 external_mini_cluster.cc:1428] Setting key 06ab9df95f8f6114106d036ae4aef66e
I20260619 20:27:43.685082 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.680028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:36973 (local address 127.21.151.62:44561)
0619 20:27:43.680069 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.680071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.680141 (+ 70us) server_negotiation.cc:407] Connection header received
0619 20:27:43.680224 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.680225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.680237 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.680295 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:43.680717 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.681167 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.681860 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.682062 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.682142 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.682907 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:43.682908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:43.682908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:43.682915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:43.683965 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.684300 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.684301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.684301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.684354 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.684541 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.684541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.684542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.684704 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:43.684762 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.684926 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.684995 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:27:43.685092 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.679957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44561 (local address 127.21.151.2:36973)
0619 20:27:43.680089 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:43.680099 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:43.680156 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:43.680424 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:43.680425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:43.680573 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:43.680635 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.680636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.681258 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.681259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.681765 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.681767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.682145 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.682146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.682208 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.682564 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:43.682568 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:43.682810 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:43.684060 (+ 1250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.684060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.684061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.684220 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.684415 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.684416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.684417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.684471 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.684789 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:43.684790 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:43.684829 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 20:27:43.684902 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:27:43.684996 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab.
Entry for principal oryx/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab.
I20260619 20:27:43.707434 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44561
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab
--principal=oryx/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:43.785673 30003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:43.785899 30003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:43.785938 30003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:43.787050 30003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:43.787117 30003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:43.787151 30003 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 20:27:43.787189 30003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:43.788632 30003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab
--principal=oryx/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.151.62:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:43.788904 30003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:43.789121 30003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:43.793833 30009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:43.793839 30011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:43.793944 30008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:43.794040 30003 server_base.cc:1034] running on GCE node
I20260619 20:27:43.794255 30003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:43.794480 30003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:43.795673 30003 hybrid_clock.cc:648] HybridClock initialized: now 1781900863795645 us; error 33 us; skew 500 ppm
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:43.798089 30003 init.cc:377] Logged in from keytab as oryx/127.21.151.3@KRBTEST.COM (short username oryx)
I20260619 20:27:43.798518 30003 webserver.cc:460] Webserver started at http://127.21.151.3:36949/ using document root <none> and password file <none>
I20260619 20:27:43.798734 30003 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:43.798795 30003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:43.798928 30003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:43.799943 30003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "57d4cf0a03834a0e868042b90cf8b9a9"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "b722767cb1e2b7a2e6034d120b805299"
server_key_iv: "d811ff6c466717202b34d02070ba1b1e"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.800283 30003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "57d4cf0a03834a0e868042b90cf8b9a9"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "b722767cb1e2b7a2e6034d120b805299"
server_key_iv: "d811ff6c466717202b34d02070ba1b1e"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.801537 30003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:43.802210 30018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:43.802368 30003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:43.802464 30003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "57d4cf0a03834a0e868042b90cf8b9a9"
format_stamp: "Formatted at 2026-06-19 20:27:43 on dist-test-slave-f9rk"
server_key: "b722767cb1e2b7a2e6034d120b805299"
server_key_iv: "d811ff6c466717202b34d02070ba1b1e"
server_key_version: "encryptionkey@0"
I20260619 20:27:43.802541 30003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:43.812592 30003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:43.814193 30003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:43.814360 30003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:43.814867 30003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:43.814924 30003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:43.814952 30003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:43.814970 30003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:43.822142 30003 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:42763
I20260619 20:27:43.822537 30003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:43.822218 30134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:42763 every 8 connection(s)
I20260619 20:27:43.824550 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.815186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:60147 (local address 127.21.151.62:44561)
0619 20:27:43.815244 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.815246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.815412 (+ 166us) server_negotiation.cc:407] Connection header received
0619 20:27:43.815993 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.815994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.816007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.816067 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:43.816707 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.817181 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.818166 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.818457 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.818526 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.821722 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:43.821723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:43.821724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:43.821733 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:43.823301 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.823714 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.823715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.823716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.823779 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.824060 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.824060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.824061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.824163 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:43.824212 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.824413 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.824468 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:27:43.824767 30027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.815172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44561 (local address 127.21.151.3:60147)
0619 20:27:43.815301 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:43.815339 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:43.815874 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:43.816129 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:43.816133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:43.816444 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:43.816612 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.816616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.817271 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.817272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.818058 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.818060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.818542 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.818543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.818680 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.819212 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:43.819223 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:43.821623 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:43.823367 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.823370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.823371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.823593 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.823857 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.823858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.823858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.823991 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.824216 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:43.824220 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:43.824323 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 20:27:43.824392 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:27:43.824507 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260619 20:27:43.825218 30135 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44561
I20260619 20:27:43.825297 30135 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:43.825459 30135 heartbeater.cc:511] Master 127.21.151.62:44561 requested a full tablet report, sending...
I20260619 20:27:43.825816 29647 ts_manager.cc:194] Registered new tserver with Master: 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763)
I20260619 20:27:43.826339 29647 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:60147
I20260619 20:27:43.831354 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 30003
I20260619 20:27:43.831481 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:43.831544 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.826233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:47665 (local address 127.21.151.62:44561)
0619 20:27:43.826270 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.826272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.826333 (+ 61us) server_negotiation.cc:407] Connection header received
0619 20:27:43.826409 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.826410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.826420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.826477 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:43.826775 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.827226 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.827968 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.828167 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.828246 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.829084 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:43.829085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:43.829086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:43.829092 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:43.830517 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.830832 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.830833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.830833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.830887 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.831082 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.831083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.831083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.831201 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:43.831270 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.831394 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.831461 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:43.831565 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.826150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44561 (local address 127.21.151.3:47665)
0619 20:27:43.826274 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:43.826284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:43.826340 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:43.826481 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:43.826482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:43.826650 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:43.826700 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.826701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.827316 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.827317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.827878 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.827880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.828247 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.828248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.828318 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.828731 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:43.828737 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:43.828983 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:43.830590 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.830591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.830592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.830754 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.830948 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:43.830949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:43.830949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:43.831003 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:43.831262 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:43.831263 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:43.831306 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 20:27:43.831371 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 20:27:43.831464 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260619 20:27:43.831770 22108 external_mini_cluster.cc:1428] Setting key 9d085c569bc89d88cc29673821aa78b3
I20260619 20:27:43.832706 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:27:43.853277 29717 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:43.848195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.21.151.62:44561)
0619 20:27:43.848252 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.848254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.848263 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:43.848320 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.848320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.848328 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.848374 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:43.848891 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.849320 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.850033 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.850205 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.850253 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.853170 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50280: BlockingRecv error: recv got EOF from 127.0.0.1:50280 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 20:27:43 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:43.860299 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.854155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.21.151.62:44561)
0619 20:27:43.854215 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.854217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.854283 (+ 66us) server_negotiation.cc:407] Connection header received
0619 20:27:43.854367 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.854368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.854377 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.854430 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:43.854869 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.855313 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.856018 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.856198 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.856270 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.858076 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:43.858077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:43.858077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:43.858084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:43.859218 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.859552 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.859553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.859554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.859648 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:43.859823 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:43.859824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:43.859825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:43.859937 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:43.860041 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.860141 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.860209 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:27:43.861992 29647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:43.863575 29647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:43.869804 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.866222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.21.151.1:37119)
0619 20:27:43.866393 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.866398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.866408 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:43.866444 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.866446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.866465 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.866557 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:43.866676 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.867428 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.868825 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.869401 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.869493 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.869583 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.869676 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.869711 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260619 20:27:43.869793 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.865819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37119 (local address 127.0.0.1:57600)
0619 20:27:43.866084 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:43.866093 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:43.866170 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:43.866562 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:43.866563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:43.866565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:43.866619 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.866621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.867518 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.867519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.868735 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.868737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.869497 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.869497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.869573 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.869589 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 20:27:43.869662 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:27:43.869710 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":87,"threads_started":1}
I20260619 20:27:43.870048 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.865847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:37603 (local address 127.0.0.1:54788)
0619 20:27:43.866248 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:43.866256 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:43.866302 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:43.866530 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:43.866533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:43.866535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:43.866608 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.866610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.867995 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.867997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.868827 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.868830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.869774 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.869775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.869870 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.869877 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 20:27:43.869915 (+ 38us) client_negotiation.cc:240] Negotiation successful
0619 20:27:43.869974 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":34,"threads_started":1}
I20260619 20:27:43.870186 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.865801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:42763 (local address 127.0.0.1:42642)
0619 20:27:43.866046 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:43.866087 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:43.866150 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:43.866623 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:43.866624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:43.866626 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:43.866664 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.866665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.867956 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.867957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.868968 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:43.868970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.869985 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.869986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:43.870042 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.870045 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:43.870080 (+ 35us) client_negotiation.cc:240] Negotiation successful
0619 20:27:43.870121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":9,"thread_start_us":63,"threads_started":1}
I20260619 20:27:43.870433 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.866018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.21.151.2:37603)
0619 20:27:43.866209 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.866214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.866295 (+ 81us) server_negotiation.cc:407] Connection header received
0619 20:27:43.866392 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.866394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.866413 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.866692 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:43.866756 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.867888 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.868910 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.869659 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.870167 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.870258 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.870287 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.870324 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260619 20:27:43.870558 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:43.865893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.21.151.3:42763)
0619 20:27:43.866456 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:43.866460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:43.866469 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:43.866506 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:43.866508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:43.866523 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:43.866736 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:43.866794 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.867864 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.869063 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:43.869754 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:43.870294 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:43.870393 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:43.870417 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 20:27:43.870463 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":87,"threads_started":1}
I20260619 20:27:43.870565 29780 tablet_service.cc:1467] Processing CreateTablet for tablet df30a0fcc2d644c18725025f8b60059c (DEFAULT_TABLE table=test-table [id=c05eaee93ff045f68efc5628b9e78e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:43.870949 29780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df30a0fcc2d644c18725025f8b60059c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:43.871105 29928 tablet_service.cc:1467] Processing CreateTablet for tablet df30a0fcc2d644c18725025f8b60059c (DEFAULT_TABLE table=test-table [id=c05eaee93ff045f68efc5628b9e78e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:43.871240 30069 tablet_service.cc:1467] Processing CreateTablet for tablet df30a0fcc2d644c18725025f8b60059c (DEFAULT_TABLE table=test-table [id=c05eaee93ff045f68efc5628b9e78e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:43.871412 29928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df30a0fcc2d644c18725025f8b60059c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:43.871544 30069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df30a0fcc2d644c18725025f8b60059c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:43.874662 30160 tablet_bootstrap.cc:492] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: Bootstrap starting.
I20260619 20:27:43.875118 30161 tablet_bootstrap.cc:492] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: Bootstrap starting.
I20260619 20:27:43.875128 30162 tablet_bootstrap.cc:492] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: Bootstrap starting.
I20260619 20:27:43.875530 30160 tablet_bootstrap.cc:654] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:43.875722 30161 tablet_bootstrap.cc:654] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:43.875792 30162 tablet_bootstrap.cc:654] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:43.875931 30160 log.cc:824] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:43.876039 30161 log.cc:824] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:43.876111 30162 log.cc:824] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:43.876829 30161 tablet_bootstrap.cc:492] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: No bootstrap required, opened a new log
I20260619 20:27:43.876878 30160 tablet_bootstrap.cc:492] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: No bootstrap required, opened a new log
I20260619 20:27:43.876911 30162 tablet_bootstrap.cc:492] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: No bootstrap required, opened a new log
I20260619 20:27:43.876914 30161 ts_tablet_manager.cc:1397] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:43.876953 30160 ts_tablet_manager.cc:1397] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 20:27:43.876972 30162 ts_tablet_manager.cc:1397] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:43.878374 30161 raft_consensus.cc:348] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } }
I20260619 20:27:43.878526 30161 raft_consensus.cc:374] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:43.878567 30161 raft_consensus.cc:729] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ec625d85604a7aa35181ce9c479292, State: Initialized, Role: FOLLOWER
I20260619 20:27:43.878674 30161 consensus_queue.cc:260] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } }
I20260619 20:27:43.878927 30161 ts_tablet_manager.cc:1428] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:27:43.878935 29853 heartbeater.cc:503] Master 127.21.151.62:44561 was elected leader, sending a full tablet report...
I20260619 20:27:43.878978 30162 raft_consensus.cc:348] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } }
I20260619 20:27:43.878978 30160 raft_consensus.cc:348] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } }
I20260619 20:27:43.879149 30160 raft_consensus.cc:374] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:43.879149 30162 raft_consensus.cc:374] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:43.879192 30160 raft_consensus.cc:729] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08126e498c2d4ae0b776ad4f138c2f37, State: Initialized, Role: FOLLOWER
I20260619 20:27:43.879192 30162 raft_consensus.cc:729] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d4cf0a03834a0e868042b90cf8b9a9, State: Initialized, Role: FOLLOWER
I20260619 20:27:43.879316 30162 consensus_queue.cc:260] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } }
I20260619 20:27:43.879316 30160 consensus_queue.cc:260] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } }
I20260619 20:27:43.879587 30160 ts_tablet_manager.cc:1428] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:27:43.879606 29994 heartbeater.cc:503] Master 127.21.151.62:44561 was elected leader, sending a full tablet report...
I20260619 20:27:43.879963 30135 heartbeater.cc:503] Master 127.21.151.62:44561 was elected leader, sending a full tablet report...
I20260619 20:27:43.880211 30162 ts_tablet_manager.cc:1428] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260619 20:27:43.926250 29995 tablet.cc:2369] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:27:44.027364 29854 tablet.cc:2369] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:27:44.073243 30136 tablet.cc:2369] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:27:44.142787 30166 raft_consensus.cc:482] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:44.142911 30166 raft_consensus.cc:504] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } }
I20260619 20:27:44.143235 30166 leader_election.cc:288] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763), 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603)
I20260619 20:27:44.147192 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.143629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:45427 (local address 127.21.151.2:37603)
0619 20:27:44.143667 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.143669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.143680 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:27:44.143716 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.143717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.143726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.143802 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:44.143974 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.144720 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.146285 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.146830 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.146907 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.146916 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.147091 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.147133 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:27:44.147214 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.143432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:37603 (local address 127.21.151.1:45427)
0619 20:27:44.143587 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.143597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:44.143660 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:44.143822 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:44.143822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:44.143824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:44.143900 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.143901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.144825 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.144826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.146146 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.146149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.146910 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.146911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.146986 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.146991 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:27:44.147073 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:27:44.147120 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":54,"spinlock_wait_cycles":12544,"thread_start_us":58,"threads_started":1}
I20260619 20:27:44.147320 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.143543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:36757 (local address 127.21.151.3:42763)
0619 20:27:44.143590 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.143593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.143647 (+ 54us) server_negotiation.cc:407] Connection header received
0619 20:27:44.143727 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.143728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.143736 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.143802 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:44.143968 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.144735 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.146291 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.146834 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.146907 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.146916 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.147189 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.147231 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:27:44.147495 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.143432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:42763 (local address 127.21.151.1:36757)
0619 20:27:44.143561 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.143573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:44.143660 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:44.143821 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:44.143822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:44.143825 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:44.143893 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.143895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.144839 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.144839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.146146 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.146149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.146975 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.146976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.147070 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.147074 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:44.147178 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 20:27:44.147226 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":96,"thread_start_us":61,"threads_started":1}
I20260619 20:27:44.147511 29948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df30a0fcc2d644c18725025f8b60059c" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08126e498c2d4ae0b776ad4f138c2f37" is_pre_election: true
I20260619 20:27:44.147691 29948 raft_consensus.cc:2454] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 0.
I20260619 20:27:44.147841 30089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df30a0fcc2d644c18725025f8b60059c" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" is_pre_election: true
I20260619 20:27:44.147953 29737 leader_election.cc:302] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08126e498c2d4ae0b776ad4f138c2f37, 76ec625d85604a7aa35181ce9c479292; no voters:
I20260619 20:27:44.147979 30089 raft_consensus.cc:2454] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 0.
I20260619 20:27:44.148129 30166 raft_consensus.cc:2781] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:44.148188 30166 raft_consensus.cc:482] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:44.148212 30166 raft_consensus.cc:3037] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.148890 30166 raft_consensus.cc:504] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } }
I20260619 20:27:44.149051 30166 leader_election.cc:288] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 election: Requested vote from peers 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763), 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603)
I20260619 20:27:44.149210 30089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df30a0fcc2d644c18725025f8b60059c" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d4cf0a03834a0e868042b90cf8b9a9"
I20260619 20:27:44.149216 29948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df30a0fcc2d644c18725025f8b60059c" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08126e498c2d4ae0b776ad4f138c2f37"
I20260619 20:27:44.149287 30089 raft_consensus.cc:3037] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.149287 29948 raft_consensus.cc:3037] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.150110 29948 raft_consensus.cc:2454] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 1.
I20260619 20:27:44.150110 30089 raft_consensus.cc:2454] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 1.
I20260619 20:27:44.150267 29737 leader_election.cc:302] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08126e498c2d4ae0b776ad4f138c2f37, 76ec625d85604a7aa35181ce9c479292; no voters:
I20260619 20:27:44.150388 30166 raft_consensus.cc:2781] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:44.150522 30166 raft_consensus.cc:686] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 1 LEADER]: Becoming Leader. State: Replica: 76ec625d85604a7aa35181ce9c479292, State: Running, Role: LEADER
I20260619 20:27:44.150606 30166 consensus_queue.cc:237] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } }
I20260619 20:27:44.151418 29647 catalog_manager.cc:5495] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76ec625d85604a7aa35181ce9c479292 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } health_report { overall_health: UNKNOWN } } }
I20260619 20:27:44.221539 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.219334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.21.151.1:37119)
0619 20:27:44.219396 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.219398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.219406 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:44.219454 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.219454 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.219471 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.219540 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:44.219684 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.220157 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.220870 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.221052 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.221105 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.221231 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.221323 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.221379 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.221408 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.221448 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:27:44.223392 29948 raft_consensus.cc:1261] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Refusing update from remote peer 76ec625d85604a7aa35181ce9c479292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:44.223424 30089 raft_consensus.cc:1261] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Refusing update from remote peer 76ec625d85604a7aa35181ce9c479292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:44.223757 30166 consensus_queue.cc:1035] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:44.223913 30171 consensus_queue.cc:1035] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:44.226668 30175 mvcc.cc:205] Tried to move back new op lower bound from 7298665939856293888 to 7298665939561472000. Current Snapshot: MvccSnapshot[applied={T|T < 7298665939856293888}]
I20260619 20:27:44.228029 30177 mvcc.cc:205] Tried to move back new op lower bound from 7298665939856293888 to 7298665939561472000. Current Snapshot: MvccSnapshot[applied={T|T < 7298665939856293888}]
I20260619 20:27:44.228997 29648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:44.229105 29648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:44.229880 29648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6: Sending DeleteTablet for 3 replicas of tablet df30a0fcc2d644c18725025f8b60059c
I20260619 20:27:44.230186 29928 tablet_service.cc:1514] Processing DeleteTablet for tablet df30a0fcc2d644c18725025f8b60059c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:54788
I20260619 20:27:44.230229 30069 tablet_service.cc:1514] Processing DeleteTablet for tablet df30a0fcc2d644c18725025f8b60059c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:42642
I20260619 20:27:44.230449 29780 tablet_service.cc:1514] Processing DeleteTablet for tablet df30a0fcc2d644c18725025f8b60059c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57600
I20260619 20:27:44.230482 30184 tablet_replica.cc:330] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: stopping tablet replica
I20260619 20:27:44.230471 29648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 20:27:44.230576 30184 raft_consensus.cc:2229] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 20:27:44.230605 29648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:44.230659 30184 raft_consensus.cc:2258] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.231035 30186 tablet_replica.cc:330] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: stopping tablet replica
I20260619 20:27:44.231144 30184 ts_tablet_manager.cc:1905] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.231402 30186 raft_consensus.cc:2229] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:44.231648 30186 raft_consensus.cc:2258] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.231938 29780 tablet_service.cc:1467] Processing CreateTablet for tablet 21773b7491344d239e8c024e7c550302 (DEFAULT_TABLE table=test-table [id=826ba01aef3d44e5aebdc00b9ee1d868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:44.232020 30069 tablet_service.cc:1467] Processing CreateTablet for tablet 21773b7491344d239e8c024e7c550302 (DEFAULT_TABLE table=test-table [id=826ba01aef3d44e5aebdc00b9ee1d868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:44.232016 29928 tablet_service.cc:1467] Processing CreateTablet for tablet 21773b7491344d239e8c024e7c550302 (DEFAULT_TABLE table=test-table [id=826ba01aef3d44e5aebdc00b9ee1d868]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:44.232049 29780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21773b7491344d239e8c024e7c550302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.232110 30069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21773b7491344d239e8c024e7c550302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.232208 29928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21773b7491344d239e8c024e7c550302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.232906 30184 ts_tablet_manager.cc:1918] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:44.232971 30184 log.cc:1196] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/df30a0fcc2d644c18725025f8b60059c
I20260619 20:27:44.233196 30184 ts_tablet_manager.cc:1939] T df30a0fcc2d644c18725025f8b60059c P 08126e498c2d4ae0b776ad4f138c2f37: Deleting consensus metadata
I20260619 20:27:44.233327 30162 tablet_bootstrap.cc:492] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9: Bootstrap starting.
I20260619 20:27:44.233341 30161 tablet_bootstrap.cc:492] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292: Bootstrap starting.
I20260619 20:27:44.233428 30160 tablet_bootstrap.cc:492] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37: Bootstrap starting.
I20260619 20:27:44.233748 30161 tablet_bootstrap.cc:654] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.233749 30162 tablet_bootstrap.cc:654] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.233772 30160 tablet_bootstrap.cc:654] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.234027 30186 ts_tablet_manager.cc:1905] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.234431 29632 catalog_manager.cc:4841] TS 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603): tablet df30a0fcc2d644c18725025f8b60059c (table test-table [id=c05eaee93ff045f68efc5628b9e78e53]) successfully deleted
I20260619 20:27:44.234650 30162 tablet_bootstrap.cc:492] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9: No bootstrap required, opened a new log
I20260619 20:27:44.234704 30162 ts_tablet_manager.cc:1397] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:44.234856 30162 raft_consensus.cc:348] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.234946 30162 raft_consensus.cc:374] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.234973 30162 raft_consensus.cc:729] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d4cf0a03834a0e868042b90cf8b9a9, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.235035 30162 consensus_queue.cc:260] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.235148 30162 ts_tablet_manager.cc:1428] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:44.235652 30186 ts_tablet_manager.cc:1918] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:44.235744 30186 log.cc:1196] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/df30a0fcc2d644c18725025f8b60059c
I20260619 20:27:44.235858 30185 tablet_replica.cc:330] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: stopping tablet replica
I20260619 20:27:44.236084 30185 raft_consensus.cc:2229] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:44.236184 30185 raft_consensus.cc:2258] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.236570 30161 tablet_bootstrap.cc:492] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292: No bootstrap required, opened a new log
I20260619 20:27:44.236615 30161 ts_tablet_manager.cc:1397] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 20:27:44.236773 30185 ts_tablet_manager.cc:1905] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.236780 30161 raft_consensus.cc:348] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.236850 30161 raft_consensus.cc:374] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.236873 30161 raft_consensus.cc:729] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ec625d85604a7aa35181ce9c479292, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.236931 30161 consensus_queue.cc:260] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.237035 30161 ts_tablet_manager.cc:1428] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:44.238240 30186 ts_tablet_manager.cc:1939] T df30a0fcc2d644c18725025f8b60059c P 76ec625d85604a7aa35181ce9c479292: Deleting consensus metadata
I20260619 20:27:44.238441 30185 ts_tablet_manager.cc:1918] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:44.238549 30185 log.cc:1196] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/df30a0fcc2d644c18725025f8b60059c
I20260619 20:27:44.238673 29634 catalog_manager.cc:4841] TS 76ec625d85604a7aa35181ce9c479292 (127.21.151.1:37119): tablet df30a0fcc2d644c18725025f8b60059c (table test-table [id=c05eaee93ff045f68efc5628b9e78e53]) successfully deleted
I20260619 20:27:44.238799 30185 ts_tablet_manager.cc:1939] T df30a0fcc2d644c18725025f8b60059c P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting consensus metadata
I20260619 20:27:44.239190 29635 catalog_manager.cc:4841] TS 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763): tablet df30a0fcc2d644c18725025f8b60059c (table test-table [id=c05eaee93ff045f68efc5628b9e78e53]) successfully deleted
I20260619 20:27:44.239953 30160 tablet_bootstrap.cc:492] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37: No bootstrap required, opened a new log
I20260619 20:27:44.240013 30160 ts_tablet_manager.cc:1397] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260619 20:27:44.240149 30160 raft_consensus.cc:348] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.240217 30160 raft_consensus.cc:374] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.240243 30160 raft_consensus.cc:729] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08126e498c2d4ae0b776ad4f138c2f37, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.240293 30160 consensus_queue.cc:260] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.240394 30160 ts_tablet_manager.cc:1428] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:44.241474 30167 raft_consensus.cc:482] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:44.241544 30167 raft_consensus.cc:504] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.241806 30167 leader_election.cc:288] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76ec625d85604a7aa35181ce9c479292 (127.21.151.1:37119), 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763)
I20260619 20:27:44.245390 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.241932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37119 (local address 127.21.151.2:44343)
0619 20:27:44.242160 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.242170 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:44.242220 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:44.242384 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:44.242385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:44.242387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:44.242452 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.242453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.243103 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.243104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.244316 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.244318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.245096 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.245097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.245205 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.245209 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:44.245265 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:27:44.245312 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
I20260619 20:27:44.245393 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.242036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:44343 (local address 127.21.151.1:37119)
0619 20:27:44.242073 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.242075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.242232 (+ 157us) server_negotiation.cc:407] Connection header received
0619 20:27:44.242261 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.242262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.242283 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.242338 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:44.242517 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.243020 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.244402 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.244979 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.245079 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.245087 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.245279 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.245329 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:44.245599 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.242012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:58891 (local address 127.21.151.3:42763)
0619 20:27:44.242052 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.242054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.242335 (+ 281us) server_negotiation.cc:407] Connection header received
0619 20:27:44.242386 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.242387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.242396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.242449 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:27:44.242597 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.243319 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.244605 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.245204 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.245251 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.245260 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.245489 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.245534 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:27:44.245837 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.241932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:42763 (local address 127.21.151.2:58891)
0619 20:27:44.242280 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.242286 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:44.242340 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:44.242455 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:44.242455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:44.242457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:27:44.242539 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.242541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.243403 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.243404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.244513 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.244515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.245312 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.245313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.245404 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.245408 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:27:44.245710 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 20:27:44.245752 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":9,"thread_start_us":54,"threads_started":1}
I20260619 20:27:44.246013 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21773b7491344d239e8c024e7c550302" candidate_uuid: "08126e498c2d4ae0b776ad4f138c2f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ec625d85604a7aa35181ce9c479292" is_pre_election: true
I20260619 20:27:44.246127 29807 raft_consensus.cc:2454] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08126e498c2d4ae0b776ad4f138c2f37 in term 0.
I20260619 20:27:44.246104 30089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21773b7491344d239e8c024e7c550302" candidate_uuid: "08126e498c2d4ae0b776ad4f138c2f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" is_pre_election: true
I20260619 20:27:44.246188 30089 raft_consensus.cc:2454] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08126e498c2d4ae0b776ad4f138c2f37 in term 0.
I20260619 20:27:44.246318 29880 leader_election.cc:302] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08126e498c2d4ae0b776ad4f138c2f37, 76ec625d85604a7aa35181ce9c479292; no voters:
I20260619 20:27:44.246469 30167 raft_consensus.cc:2781] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:44.246531 30167 raft_consensus.cc:482] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:44.246562 30167 raft_consensus.cc:3037] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.247308 30167 raft_consensus.cc:504] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.247432 30167 leader_election.cc:288] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [CANDIDATE]: Term 1 election: Requested vote from peers 76ec625d85604a7aa35181ce9c479292 (127.21.151.1:37119), 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763)
I20260619 20:27:44.247592 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21773b7491344d239e8c024e7c550302" candidate_uuid: "08126e498c2d4ae0b776ad4f138c2f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76ec625d85604a7aa35181ce9c479292"
I20260619 20:27:44.247637 30089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21773b7491344d239e8c024e7c550302" candidate_uuid: "08126e498c2d4ae0b776ad4f138c2f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d4cf0a03834a0e868042b90cf8b9a9"
I20260619 20:27:44.247694 30089 raft_consensus.cc:3037] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.247694 29807 raft_consensus.cc:3037] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.248252 29807 raft_consensus.cc:2454] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08126e498c2d4ae0b776ad4f138c2f37 in term 1.
I20260619 20:27:44.248337 30089 raft_consensus.cc:2454] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08126e498c2d4ae0b776ad4f138c2f37 in term 1.
I20260619 20:27:44.248415 29880 leader_election.cc:302] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08126e498c2d4ae0b776ad4f138c2f37, 76ec625d85604a7aa35181ce9c479292; no voters:
I20260619 20:27:44.248522 30167 raft_consensus.cc:2781] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:44.248656 30167 raft_consensus.cc:686] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 LEADER]: Becoming Leader. State: Replica: 08126e498c2d4ae0b776ad4f138c2f37, State: Running, Role: LEADER
I20260619 20:27:44.248719 30167 consensus_queue.cc:237] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.249321 29648 catalog_manager.cc:5495] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2). New cstate: current_term: 1 leader_uuid: "08126e498c2d4ae0b776ad4f138c2f37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } health_report { overall_health: UNKNOWN } } }
Jun 19 20:27:44 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900863, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.62@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:44.280036 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.272387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.21.151.62:44561)
0619 20:27:44.272439 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.272442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.272607 (+ 165us) server_negotiation.cc:407] Connection header received
0619 20:27:44.272637 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.272638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.272649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.272705 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:44.273159 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.273955 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.274642 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.274943 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.275023 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.277274 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:44.277276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:44.277276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:44.277283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:44.278860 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:44.279198 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:44.279199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:44.279200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:44.279260 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:44.279437 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:44.279437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:44.279438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:44.279567 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:44.279661 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.279822 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.279915 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 20:27:44.280107 30199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.272317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44561 (local address 127.0.0.1:50298)
0619 20:27:44.272476 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.272493 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:44.272561 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:44.272723 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:44.272724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:44.273027 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:44.273072 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.273074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.274037 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.274038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.274555 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.274557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.275026 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.275026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.275064 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.275609 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:44.275654 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:44.277195 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:44.278934 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:44.278940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:44.278941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:44.279126 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:44.279325 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:44.279325 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:44.279326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:44.279378 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:44.279674 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:44.279681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:44.279738 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 20:27:44.279821 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 20:27:44.279965 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":41,"thread_start_us":98,"threads_started":1}
I20260619 20:27:44.282063 29648 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:50298:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 20:27:44.282287 29648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:44.283730 29780 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7fd6c54e9f45e6925aaaea45148f17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92967f56520742b38f0b040a756816ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:27:44.283766 30069 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7fd6c54e9f45e6925aaaea45148f17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92967f56520742b38f0b040a756816ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:27:44.283883 30069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7fd6c54e9f45e6925aaaea45148f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.283892 29780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7fd6c54e9f45e6925aaaea45148f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.283870 29928 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7fd6c54e9f45e6925aaaea45148f17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=92967f56520742b38f0b040a756816ce]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 20:27:44.284021 29928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7fd6c54e9f45e6925aaaea45148f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.285305 30162 tablet_bootstrap.cc:492] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: Bootstrap starting.
I20260619 20:27:44.285398 30160 tablet_bootstrap.cc:492] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: Bootstrap starting.
I20260619 20:27:44.285697 30162 tablet_bootstrap.cc:654] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.285774 30160 tablet_bootstrap.cc:654] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.286068 30161 tablet_bootstrap.cc:492] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: Bootstrap starting.
I20260619 20:27:44.286417 30162 tablet_bootstrap.cc:492] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: No bootstrap required, opened a new log
I20260619 20:27:44.286473 30162 ts_tablet_manager.cc:1397] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:44.286484 30161 tablet_bootstrap.cc:654] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.286640 30162 raft_consensus.cc:348] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.286738 30162 raft_consensus.cc:374] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.286759 30162 raft_consensus.cc:729] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d4cf0a03834a0e868042b90cf8b9a9, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.286811 30162 consensus_queue.cc:260] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.286917 30162 ts_tablet_manager.cc:1428] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:44.287086 30168 tablet_replica.cc:439] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } }
I20260619 20:27:44.287189 30168 tablet_replica.cc:442] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:44.287256 30160 tablet_bootstrap.cc:492] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: No bootstrap required, opened a new log
I20260619 20:27:44.287298 30160 ts_tablet_manager.cc:1397] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:44.287451 30160 raft_consensus.cc:348] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.287540 30161 tablet_bootstrap.cc:492] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: No bootstrap required, opened a new log
I20260619 20:27:44.287546 30160 raft_consensus.cc:374] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.287570 30160 raft_consensus.cc:729] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08126e498c2d4ae0b776ad4f138c2f37, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.287580 30161 ts_tablet_manager.cc:1397] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:44.287637 30160 consensus_queue.cc:260] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.287739 30160 ts_tablet_manager.cc:1428] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:44.287734 30161 raft_consensus.cc:348] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.287818 30161 raft_consensus.cc:374] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.287838 30161 raft_consensus.cc:729] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ec625d85604a7aa35181ce9c479292, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.287879 30161 consensus_queue.cc:260] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.287866 30167 tablet_replica.cc:439] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } }
I20260619 20:27:44.287930 30167 tablet_replica.cc:442] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:44.287986 30161 ts_tablet_manager.cc:1428] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:44.288091 30181 tablet_replica.cc:439] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } }
I20260619 20:27:44.288182 30181 tablet_replica.cc:442] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:44.289573 30181 raft_consensus.cc:482] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:44.289623 30181 raft_consensus.cc:504] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.289757 30181 leader_election.cc:288] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603), 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763)
I20260619 20:27:44.289876 29948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7fd6c54e9f45e6925aaaea45148f17" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08126e498c2d4ae0b776ad4f138c2f37" is_pre_election: true
I20260619 20:27:44.289937 29948 raft_consensus.cc:2454] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 0.
I20260619 20:27:44.289909 30089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7fd6c54e9f45e6925aaaea45148f17" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" is_pre_election: true
I20260619 20:27:44.289991 30089 raft_consensus.cc:2454] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 0.
I20260619 20:27:44.290064 29737 leader_election.cc:302] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08126e498c2d4ae0b776ad4f138c2f37, 76ec625d85604a7aa35181ce9c479292; no voters:
I20260619 20:27:44.290143 30181 raft_consensus.cc:2781] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:44.290194 30181 raft_consensus.cc:482] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:44.290217 30181 raft_consensus.cc:3037] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.290845 30181 raft_consensus.cc:504] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.290978 30181 leader_election.cc:288] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 election: Requested vote from peers 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603), 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763)
I20260619 20:27:44.291103 29948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7fd6c54e9f45e6925aaaea45148f17" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08126e498c2d4ae0b776ad4f138c2f37"
I20260619 20:27:44.291167 29948 raft_consensus.cc:3037] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.291146 30089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7fd6c54e9f45e6925aaaea45148f17" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d4cf0a03834a0e868042b90cf8b9a9"
I20260619 20:27:44.291210 30089 raft_consensus.cc:3037] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.291765 29948 raft_consensus.cc:2454] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 1.
I20260619 20:27:44.291869 30089 raft_consensus.cc:2454] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 1.
I20260619 20:27:44.291913 29737 leader_election.cc:302] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08126e498c2d4ae0b776ad4f138c2f37, 76ec625d85604a7aa35181ce9c479292; no voters:
I20260619 20:27:44.292002 30181 raft_consensus.cc:2781] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:44.292053 30181 raft_consensus.cc:686] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [term 1 LEADER]: Becoming Leader. State: Replica: 76ec625d85604a7aa35181ce9c479292, State: Running, Role: LEADER
I20260619 20:27:44.292101 30181 consensus_queue.cc:237] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.292412 30181 tablet_replica.cc:439] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: TxnStatusTablet state changed. Reason: New leader 76ec625d85604a7aa35181ce9c479292. Latest consensus state: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } }
I20260619 20:27:44.292486 30181 tablet_replica.cc:442] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:27:44.292649 29648 catalog_manager.cc:5495] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76ec625d85604a7aa35181ce9c479292 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } health_report { overall_health: UNKNOWN } } }
I20260619 20:27:44.293066 30204 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 20:27:44.293169 30204 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 20:27:44.300241 30199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.297657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37119 (local address 127.0.0.1:57612)
0619 20:27:44.297719 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.297729 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:44.297781 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:44.297922 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:44.297923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:44.297924 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:44.297973 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.297975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.298856 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.298856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.299412 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.299414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.299874 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.299875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.299931 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.299944 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.300121 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.300124 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:27:44.300155 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 20:27:44.300189 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 20:27:44.300244 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.297681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.21.151.1:37119)
0619 20:27:44.297721 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.297723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.297767 (+ 44us) server_negotiation.cc:407] Connection header received
0619 20:27:44.297852 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.297852 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.297862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.297918 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:44.298022 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.298777 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.299510 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.299798 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.299875 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.300022 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.300058 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.300122 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.300158 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.300180 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:27:44.301316 30089 raft_consensus.cc:1261] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Refusing update from remote peer 76ec625d85604a7aa35181ce9c479292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:44.301339 29948 raft_consensus.cc:1261] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Refusing update from remote peer 76ec625d85604a7aa35181ce9c479292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:44.301564 30166 consensus_queue.cc:1035] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:44.301674 30181 consensus_queue.cc:1035] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:44.302085 30168 tablet_replica.cc:439] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: TxnStatusTablet state changed. Reason: New leader 76ec625d85604a7aa35181ce9c479292. Latest consensus state: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } }
I20260619 20:27:44.302215 30168 tablet_replica.cc:442] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:44.302291 30167 tablet_replica.cc:439] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: TxnStatusTablet state changed. Reason: New leader 76ec625d85604a7aa35181ce9c479292. Latest consensus state: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } }
I20260619 20:27:44.302361 30167 tablet_replica.cc:442] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:44.302922 30166 tablet_replica.cc:439] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } }
I20260619 20:27:44.303023 30166 tablet_replica.cc:442] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:27:44.303076 30166 tablet_replica.cc:439] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } }
I20260619 20:27:44.303119 30166 tablet_replica.cc:442] T 1d7fd6c54e9f45e6925aaaea45148f17 P 76ec625d85604a7aa35181ce9c479292: This TxnStatusTablet replica's current role is: LEADER
I20260619 20:27:44.303138 30167 tablet_replica.cc:439] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } }
I20260619 20:27:44.303220 30167 tablet_replica.cc:442] T 1d7fd6c54e9f45e6925aaaea45148f17 P 08126e498c2d4ae0b776ad4f138c2f37: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:44.303267 30168 tablet_replica.cc:439] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } } }
I20260619 20:27:44.303336 30168 tablet_replica.cc:442] T 1d7fd6c54e9f45e6925aaaea45148f17 P 57d4cf0a03834a0e868042b90cf8b9a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 20:27:44.304003 30174 mvcc.cc:205] Tried to move back new op lower bound from 7298665940176777216 to 7298665940140949504. Current Snapshot: MvccSnapshot[applied={T|T < 7298665940176777216 or (T in {7298665940176777216})}]
I20260619 20:27:44.307826 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.305265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.21.151.2:37603)
0619 20:27:44.305337 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.305339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.305347 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:44.305380 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.305380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.305398 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.305535 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:44.305649 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.306123 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.306957 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.307162 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.307231 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.307404 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.307520 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.307603 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.307716 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.307739 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 20:27:44.313300 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.310838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:49259 (local address 127.21.151.1:37119)
0619 20:27:44.310884 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.310886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.311015 (+ 129us) server_negotiation.cc:407] Connection header received
0619 20:27:44.311078 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.311079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.311089 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.311159 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:44.311311 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.311834 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.312589 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.312791 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.312871 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.313052 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.313085 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.313158 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.313217 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.313233 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:27:44.313519 30210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.310788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37119 (local address 127.21.151.2:49259)
0619 20:27:44.310935 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.310945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:44.311030 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:44.311159 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:44.311160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:44.311161 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:44.311220 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.311221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.311910 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.311911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.312490 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.312492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.312873 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.312874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.312935 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.312956 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.313159 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.313161 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:27:44.313212 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:27:44.313252 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260619 20:27:44.315903 29807 raft_consensus.cc:1261] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Refusing update from remote peer 08126e498c2d4ae0b776ad4f138c2f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:44.316141 30089 raft_consensus.cc:1261] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Refusing update from remote peer 08126e498c2d4ae0b776ad4f138c2f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:44.316182 30167 consensus_queue.cc:1035] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:44.316344 30192 consensus_queue.cc:1035] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:44.324669 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.321829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:53197 (local address 127.21.151.2:37603)
0619 20:27:44.321875 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.321877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.322033 (+ 156us) server_negotiation.cc:407] Connection header received
0619 20:27:44.322078 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.322079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.322090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.322156 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:44.322238 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.322939 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.323965 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.324238 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.324317 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.324440 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.324473 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.324530 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.324583 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.324602 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 20:27:44.324692 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.321645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:37603 (local address 127.21.151.1:53197)
0619 20:27:44.321960 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.321970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:44.322039 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:44.322157 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:44.322157 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:44.322159 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:44.322203 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.322204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.323029 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.323029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.323880 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.323882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.324313 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.324314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.324358 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.324370 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.324540 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.324542 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:27:44.324578 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 20:27:44.324613 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":89,"threads_started":1}
I20260619 20:27:44.334299 29648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:44.334411 29648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:44.335014 29648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6: Sending DeleteTablet for 3 replicas of tablet 21773b7491344d239e8c024e7c550302
I20260619 20:27:44.335255 29928 tablet_service.cc:1514] Processing DeleteTablet for tablet 21773b7491344d239e8c024e7c550302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:54788
I20260619 20:27:44.335341 30184 tablet_replica.cc:330] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37: stopping tablet replica
I20260619 20:27:44.335345 29780 tablet_service.cc:1514] Processing DeleteTablet for tablet 21773b7491344d239e8c024e7c550302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57600
I20260619 20:27:44.335383 30184 raft_consensus.cc:2229] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:44.335427 30186 tablet_replica.cc:330] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292: stopping tablet replica
I20260619 20:27:44.335457 30184 raft_consensus.cc:2258] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.335470 30186 raft_consensus.cc:2229] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:44.335515 30186 raft_consensus.cc:2258] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.335976 30184 ts_tablet_manager.cc:1905] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.337754 30184 ts_tablet_manager.cc:1918] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:44.337808 30184 log.cc:1196] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/21773b7491344d239e8c024e7c550302
I20260619 20:27:44.338043 30184 ts_tablet_manager.cc:1939] T 21773b7491344d239e8c024e7c550302 P 08126e498c2d4ae0b776ad4f138c2f37: Deleting consensus metadata
I20260619 20:27:44.338335 29632 catalog_manager.cc:4841] TS 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603): tablet 21773b7491344d239e8c024e7c550302 (table test-table [id=826ba01aef3d44e5aebdc00b9ee1d868]) successfully deleted
I20260619 20:27:44.338672 30186 ts_tablet_manager.cc:1905] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.338764 30069 tablet_service.cc:1514] Processing DeleteTablet for tablet 21773b7491344d239e8c024e7c550302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:42642
I20260619 20:27:44.339708 30185 tablet_replica.cc:330] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9: stopping tablet replica
I20260619 20:27:44.339772 30185 raft_consensus.cc:2229] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:44.339807 30185 raft_consensus.cc:2258] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.340220 30185 ts_tablet_manager.cc:1905] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.340220 30186 ts_tablet_manager.cc:1918] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:44.340276 30186 log.cc:1196] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/21773b7491344d239e8c024e7c550302
I20260619 20:27:44.340487 30186 ts_tablet_manager.cc:1939] T 21773b7491344d239e8c024e7c550302 P 76ec625d85604a7aa35181ce9c479292: Deleting consensus metadata
I20260619 20:27:44.341780 30185 ts_tablet_manager.cc:1918] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:44.341831 30185 log.cc:1196] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/21773b7491344d239e8c024e7c550302
I20260619 20:27:44.342036 30185 ts_tablet_manager.cc:1939] T 21773b7491344d239e8c024e7c550302 P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting consensus metadata
I20260619 20:27:44.342293 29634 catalog_manager.cc:4841] TS 76ec625d85604a7aa35181ce9c479292 (127.21.151.1:37119): tablet 21773b7491344d239e8c024e7c550302 (table test-table [id=826ba01aef3d44e5aebdc00b9ee1d868]) successfully deleted
I20260619 20:27:44.342599 29635 catalog_manager.cc:4841] TS 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763): tablet 21773b7491344d239e8c024e7c550302 (table test-table [id=826ba01aef3d44e5aebdc00b9ee1d868]) successfully deleted
Jun 19 20:27:44 dist-test-slave-f9rk krb5kdc[29603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 20:27:44 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:44 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:27:44.370972 29717 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:44.364240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.21.151.62:44561)
0619 20:27:44.364287 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.364289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.364463 (+ 174us) server_negotiation.cc:407] Connection header received
0619 20:27:44.364526 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.364527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.364540 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.364613 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:44.365334 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.366130 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.367200 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.367520 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.367591 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.370896 (+ 3305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50312: BlockingRecv error: recv got EOF from 127.0.0.1:50312 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 20:27:44 dist-test-slave-f9rk krb5kdc[29603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM
I20260619 20:27:44.380722 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.371859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.21.151.62:44561)
0619 20:27:44.371893 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.371895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.372115 (+ 220us) server_negotiation.cc:407] Connection header received
0619 20:27:44.372178 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.372179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.372190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.372247 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:44.372787 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.373557 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.374637 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.374891 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.374980 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.377649 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:44.377650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:44.377651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:44.377658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:44.379337 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:44.379766 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:44.379767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:44.379767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:44.379828 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:44.380125 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:44.380125 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:44.380126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:44.380255 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:44.380353 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.380536 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.380626 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:27:44.382247 29648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:44.382411 29648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:44.383747 29928 tablet_service.cc:1467] Processing CreateTablet for tablet 24884230976143d4aaa603ac90841dfa (DEFAULT_TABLE table=test-table [id=873f7ee5ad43449a883336423c8bed26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:44.383877 29928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24884230976143d4aaa603ac90841dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.383850 29780 tablet_service.cc:1467] Processing CreateTablet for tablet 24884230976143d4aaa603ac90841dfa (DEFAULT_TABLE table=test-table [id=873f7ee5ad43449a883336423c8bed26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:44.383980 29780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24884230976143d4aaa603ac90841dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.385166 30160 tablet_bootstrap.cc:492] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37: Bootstrap starting.
I20260619 20:27:44.385180 30161 tablet_bootstrap.cc:492] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292: Bootstrap starting.
I20260619 20:27:44.385234 30069 tablet_service.cc:1467] Processing CreateTablet for tablet 24884230976143d4aaa603ac90841dfa (DEFAULT_TABLE table=test-table [id=873f7ee5ad43449a883336423c8bed26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:44.385345 30069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24884230976143d4aaa603ac90841dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.385490 30161 tablet_bootstrap.cc:654] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.385564 30160 tablet_bootstrap.cc:654] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.386485 30160 tablet_bootstrap.cc:492] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37: No bootstrap required, opened a new log
I20260619 20:27:44.386540 30160 ts_tablet_manager.cc:1397] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:44.386586 30162 tablet_bootstrap.cc:492] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: Bootstrap starting.
I20260619 20:27:44.386682 30160 raft_consensus.cc:348] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.386758 30160 raft_consensus.cc:374] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.386782 30160 raft_consensus.cc:729] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08126e498c2d4ae0b776ad4f138c2f37, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.386826 30160 consensus_queue.cc:260] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.386911 30160 ts_tablet_manager.cc:1428] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:44.386969 30162 tablet_bootstrap.cc:654] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.387382 30161 tablet_bootstrap.cc:492] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292: No bootstrap required, opened a new log
I20260619 20:27:44.387428 30161 ts_tablet_manager.cc:1397] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:44.387544 30161 raft_consensus.cc:348] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.387601 30161 raft_consensus.cc:374] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.387640 30161 raft_consensus.cc:729] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ec625d85604a7aa35181ce9c479292, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.387677 30161 consensus_queue.cc:260] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.387749 30161 ts_tablet_manager.cc:1428] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:44.388590 30162 tablet_bootstrap.cc:492] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: No bootstrap required, opened a new log
I20260619 20:27:44.388639 30162 ts_tablet_manager.cc:1397] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:27:44.388782 30162 raft_consensus.cc:348] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.388840 30162 raft_consensus.cc:374] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.388860 30162 raft_consensus.cc:729] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d4cf0a03834a0e868042b90cf8b9a9, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.388898 30162 consensus_queue.cc:260] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.388988 30162 ts_tablet_manager.cc:1428] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:44.410096 30166 raft_consensus.cc:482] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:44.410185 30166 raft_consensus.cc:504] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.410318 30166 leader_election.cc:288] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603), 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763)
I20260619 20:27:44.410501 29948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24884230976143d4aaa603ac90841dfa" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08126e498c2d4ae0b776ad4f138c2f37" is_pre_election: true
I20260619 20:27:44.410591 29948 raft_consensus.cc:2454] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 0.
I20260619 20:27:44.410563 30089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24884230976143d4aaa603ac90841dfa" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" is_pre_election: true
I20260619 20:27:44.410666 30089 raft_consensus.cc:2454] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 0.
I20260619 20:27:44.410739 29737 leader_election.cc:302] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08126e498c2d4ae0b776ad4f138c2f37, 76ec625d85604a7aa35181ce9c479292; no voters:
I20260619 20:27:44.410804 30166 raft_consensus.cc:2781] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:44.410825 30166 raft_consensus.cc:482] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:44.410841 30166 raft_consensus.cc:3037] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.411597 30166 raft_consensus.cc:504] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.411741 30166 leader_election.cc:288] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 election: Requested vote from peers 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603), 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763)
I20260619 20:27:44.411926 29948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24884230976143d4aaa603ac90841dfa" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08126e498c2d4ae0b776ad4f138c2f37"
I20260619 20:27:44.411999 29948 raft_consensus.cc:3037] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.411971 30089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24884230976143d4aaa603ac90841dfa" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d4cf0a03834a0e868042b90cf8b9a9"
I20260619 20:27:44.412032 30089 raft_consensus.cc:3037] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.412698 29948 raft_consensus.cc:2454] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 1.
I20260619 20:27:44.412801 30089 raft_consensus.cc:2454] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 1.
I20260619 20:27:44.412843 29737 leader_election.cc:302] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08126e498c2d4ae0b776ad4f138c2f37, 76ec625d85604a7aa35181ce9c479292; no voters:
I20260619 20:27:44.413053 30166 raft_consensus.cc:2781] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:44.413100 30166 raft_consensus.cc:686] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 1 LEADER]: Becoming Leader. State: Replica: 76ec625d85604a7aa35181ce9c479292, State: Running, Role: LEADER
I20260619 20:27:44.413141 30166 consensus_queue.cc:237] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.413606 29647 catalog_manager.cc:5495] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76ec625d85604a7aa35181ce9c479292 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } health_report { overall_health: UNKNOWN } } }
I20260619 20:27:44.426036 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.422816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.21.151.1:37119)
0619 20:27:44.422866 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.422868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.422933 (+ 65us) server_negotiation.cc:407] Connection header received
0619 20:27:44.423011 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.423012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.423023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.423084 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:44.423234 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.423971 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.425118 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.425414 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.425490 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.425679 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.425715 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.425799 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.425938 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.425964 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:27:44.426723 29948 raft_consensus.cc:1261] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Refusing update from remote peer 76ec625d85604a7aa35181ce9c479292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:44.426767 30089 raft_consensus.cc:1261] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Refusing update from remote peer 76ec625d85604a7aa35181ce9c479292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:44.426971 30181 consensus_queue.cc:1035] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:44.427074 30166 consensus_queue.cc:1035] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:44.429370 29648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:44.429471 29648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:44.430173 29648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6: Sending DeleteTablet for 3 replicas of tablet 24884230976143d4aaa603ac90841dfa
I20260619 20:27:44.430676 29928 tablet_service.cc:1514] Processing DeleteTablet for tablet 24884230976143d4aaa603ac90841dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:54788
I20260619 20:27:44.430759 30184 tablet_replica.cc:330] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37: stopping tablet replica
I20260619 20:27:44.430792 30184 raft_consensus.cc:2229] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:44.430776 30069 tablet_service.cc:1514] Processing DeleteTablet for tablet 24884230976143d4aaa603ac90841dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:42642
I20260619 20:27:44.430824 30184 raft_consensus.cc:2258] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.430862 30185 tablet_replica.cc:330] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: stopping tablet replica
I20260619 20:27:44.430907 30185 raft_consensus.cc:2229] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:44.430939 30185 pending_rounds.cc:62] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: Trying to abort 2 pending ops.
I20260619 20:27:44.430956 30185 pending_rounds.cc:68] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298665940690305024 op_type: WRITE_OP write_request { tablet_id: "24884230976143d4aaa603ac90841dfa" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\354\316\326\321\006\"8\n\ntest-admin\022*\n 873f7ee5ad43449a883336423c8bed26\020\001\030\001 \001(\001" signature: "<redacted>""P\253\212\347\373\343\311\355\354[>\224!_x\300\004\326\005|@\004^\336Y\256 \004\026\277\335\253(\273\305\257\223\301\220\3407\273\013\000\375\227B\301%\252e\377E\206\336\003\222P\360\255\343\345\232\252" signing_key_seq_num: 0 } } request_id { client_id: "c95fbfa399b24dcc996312f18ce01131" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260619 20:27:44.431118 30185 pending_rounds.cc:68] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298665940636426240 op_type: NO_OP noop_request { }
I20260619 20:27:44.431159 30185 raft_consensus.cc:2866] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260619 20:27:44.431185 30185 raft_consensus.cc:2258] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.431726 29647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:27:44.431865 29647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:27:44.432435 29780 tablet_service.cc:1514] Processing DeleteTablet for tablet 24884230976143d4aaa603ac90841dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57600
I20260619 20:27:44.432523 30186 tablet_replica.cc:330] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292: stopping tablet replica
I20260619 20:27:44.432564 30186 raft_consensus.cc:2229] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:44.432611 30186 raft_consensus.cc:2258] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.433197 30186 ts_tablet_manager.cc:1905] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.434741 30186 ts_tablet_manager.cc:1918] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:44.434808 30186 log.cc:1196] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/24884230976143d4aaa603ac90841dfa
I20260619 20:27:44.435040 30186 ts_tablet_manager.cc:1939] T 24884230976143d4aaa603ac90841dfa P 76ec625d85604a7aa35181ce9c479292: Deleting consensus metadata
I20260619 20:27:44.435387 29634 catalog_manager.cc:4841] TS 76ec625d85604a7aa35181ce9c479292 (127.21.151.1:37119): tablet 24884230976143d4aaa603ac90841dfa (table test-table [id=873f7ee5ad43449a883336423c8bed26]) successfully deleted
I20260619 20:27:44.435797 30185 ts_tablet_manager.cc:1905] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.436077 29928 tablet_service.cc:1467] Processing CreateTablet for tablet 02414b168fec4fa6aaec89e253356f6b (DEFAULT_TABLE table=test-table [id=5e1c43b98acc498b93d9e2f26bcd5d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:44.436187 29928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02414b168fec4fa6aaec89e253356f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.437145 30185 ts_tablet_manager.cc:1918] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:44.437193 30185 log.cc:1196] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/24884230976143d4aaa603ac90841dfa
I20260619 20:27:44.437389 30185 ts_tablet_manager.cc:1939] T 24884230976143d4aaa603ac90841dfa P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting consensus metadata
I20260619 20:27:44.437463 30160 tablet_bootstrap.cc:492] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37: Bootstrap starting.
I20260619 20:27:44.437986 30160 tablet_bootstrap.cc:654] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.438476 29780 tablet_service.cc:1467] Processing CreateTablet for tablet 02414b168fec4fa6aaec89e253356f6b (DEFAULT_TABLE table=test-table [id=5e1c43b98acc498b93d9e2f26bcd5d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:44.438596 29780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02414b168fec4fa6aaec89e253356f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.439836 30161 tablet_bootstrap.cc:492] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292: Bootstrap starting.
I20260619 20:27:44.440086 30184 ts_tablet_manager.cc:1905] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.440124 30161 tablet_bootstrap.cc:654] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.441450 30184 ts_tablet_manager.cc:1918] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:27:44.441512 30184 log.cc:1196] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/24884230976143d4aaa603ac90841dfa
I20260619 20:27:44.441732 30184 ts_tablet_manager.cc:1939] T 24884230976143d4aaa603ac90841dfa P 08126e498c2d4ae0b776ad4f138c2f37: Deleting consensus metadata
I20260619 20:27:44.442063 29632 catalog_manager.cc:4841] TS 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603): tablet 24884230976143d4aaa603ac90841dfa (table test-table [id=873f7ee5ad43449a883336423c8bed26]) successfully deleted
I20260619 20:27:44.442266 30069 tablet_service.cc:1467] Processing CreateTablet for tablet 02414b168fec4fa6aaec89e253356f6b (DEFAULT_TABLE table=test-table [id=5e1c43b98acc498b93d9e2f26bcd5d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:27:44.442369 30069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02414b168fec4fa6aaec89e253356f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:44.442605 30160 tablet_bootstrap.cc:492] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37: No bootstrap required, opened a new log
I20260619 20:27:44.442646 30160 ts_tablet_manager.cc:1397] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 20:27:44.442783 30160 raft_consensus.cc:348] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.442850 30160 raft_consensus.cc:374] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.442870 30160 raft_consensus.cc:729] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08126e498c2d4ae0b776ad4f138c2f37, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.442929 30160 consensus_queue.cc:260] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.443010 30160 ts_tablet_manager.cc:1428] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:44.443300 29635 catalog_manager.cc:4841] TS 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763): tablet 24884230976143d4aaa603ac90841dfa (table test-table [id=873f7ee5ad43449a883336423c8bed26]) successfully deleted
I20260619 20:27:44.443564 30162 tablet_bootstrap.cc:492] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9: Bootstrap starting.
I20260619 20:27:44.444198 30162 tablet_bootstrap.cc:654] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:44.444273 30161 tablet_bootstrap.cc:492] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292: No bootstrap required, opened a new log
I20260619 20:27:44.444317 30161 ts_tablet_manager.cc:1397] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 20:27:44.444422 30161 raft_consensus.cc:348] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.444483 30161 raft_consensus.cc:374] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.444502 30161 raft_consensus.cc:729] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ec625d85604a7aa35181ce9c479292, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.444542 30161 consensus_queue.cc:260] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.444658 30161 ts_tablet_manager.cc:1428] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:44.444860 30162 tablet_bootstrap.cc:492] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9: No bootstrap required, opened a new log
I20260619 20:27:44.444898 30162 ts_tablet_manager.cc:1397] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:44.445032 30162 raft_consensus.cc:348] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.445103 30162 raft_consensus.cc:374] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:44.445120 30162 raft_consensus.cc:729] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d4cf0a03834a0e868042b90cf8b9a9, State: Initialized, Role: FOLLOWER
I20260619 20:27:44.445154 30162 consensus_queue.cc:260] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.445218 30162 ts_tablet_manager.cc:1428] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:44.755863 30166 raft_consensus.cc:482] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:27:44.755965 30166 raft_consensus.cc:504] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.756138 30166 leader_election.cc:288] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603), 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763)
I20260619 20:27:44.756343 30089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02414b168fec4fa6aaec89e253356f6b" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" is_pre_election: true
I20260619 20:27:44.756384 29948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02414b168fec4fa6aaec89e253356f6b" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08126e498c2d4ae0b776ad4f138c2f37" is_pre_election: true
I20260619 20:27:44.756456 29948 raft_consensus.cc:2454] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 0.
I20260619 20:27:44.756484 30089 raft_consensus.cc:2454] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 0.
I20260619 20:27:44.756637 29737 leader_election.cc:302] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08126e498c2d4ae0b776ad4f138c2f37, 76ec625d85604a7aa35181ce9c479292; no voters:
I20260619 20:27:44.756742 30166 raft_consensus.cc:2781] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:27:44.756786 30166 raft_consensus.cc:482] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:27:44.756800 30166 raft_consensus.cc:3037] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.757417 30166 raft_consensus.cc:504] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.757565 30166 leader_election.cc:288] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 election: Requested vote from peers 08126e498c2d4ae0b776ad4f138c2f37 (127.21.151.2:37603), 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763)
I20260619 20:27:44.757709 29948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02414b168fec4fa6aaec89e253356f6b" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08126e498c2d4ae0b776ad4f138c2f37"
I20260619 20:27:44.757714 30089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02414b168fec4fa6aaec89e253356f6b" candidate_uuid: "76ec625d85604a7aa35181ce9c479292" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d4cf0a03834a0e868042b90cf8b9a9"
I20260619 20:27:44.757790 29948 raft_consensus.cc:3037] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.757803 30089 raft_consensus.cc:3037] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:44.758389 29948 raft_consensus.cc:2454] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 1.
I20260619 20:27:44.758404 30089 raft_consensus.cc:2454] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76ec625d85604a7aa35181ce9c479292 in term 1.
I20260619 20:27:44.758556 29737 leader_election.cc:302] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08126e498c2d4ae0b776ad4f138c2f37, 76ec625d85604a7aa35181ce9c479292; no voters:
I20260619 20:27:44.758687 30166 raft_consensus.cc:2781] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:44.758762 30166 raft_consensus.cc:686] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 1 LEADER]: Becoming Leader. State: Replica: 76ec625d85604a7aa35181ce9c479292, State: Running, Role: LEADER
I20260619 20:27:44.758810 30166 consensus_queue.cc:237] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } }
I20260619 20:27:44.759375 29648 catalog_manager.cc:5495] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76ec625d85604a7aa35181ce9c479292 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "76ec625d85604a7aa35181ce9c479292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76ec625d85604a7aa35181ce9c479292" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 } health_report { overall_health: UNKNOWN } } }
I20260619 20:27:44.800887 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.798930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:52239 (local address 127.21.151.1:37119)
0619 20:27:44.798970 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:44.798972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:44.798994 (+ 22us) server_negotiation.cc:407] Connection header received
0619 20:27:44.799011 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:44.799012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:44.799020 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:44.799057 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:27:44.799134 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.799580 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.800303 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.800478 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.800528 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.800683 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.800706 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.800744 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:44.800793 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 20:27:44.800812 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20}
I20260619 20:27:44.800895 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.798858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37119 (local address 127.21.151.1:52239)
0619 20:27:44.798911 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.798924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:44.798964 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:44.799059 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:44.799060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:44.799061 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 20:27:44.799098 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.799099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.799691 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.799692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.800237 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:44.800238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:44.800551 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:44.800552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:44.800617 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:44.800623 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 20:27:44.800760 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 20:27:44.800761 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 20:27:44.800792 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 20:27:44.800833 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:27:44.802614 30089 raft_consensus.cc:1261] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Refusing update from remote peer 76ec625d85604a7aa35181ce9c479292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:44.802614 29948 raft_consensus.cc:1261] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Refusing update from remote peer 76ec625d85604a7aa35181ce9c479292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:27:44.802820 30171 consensus_queue.cc:1035] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d4cf0a03834a0e868042b90cf8b9a9" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:44.802919 30181 consensus_queue.cc:1035] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08126e498c2d4ae0b776ad4f138c2f37" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 37603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:27:44.812599 29648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:44.812697 29648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:27:44.813254 29648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dafe6cf95014fa09b1d34d89d2594e6: Sending DeleteTablet for 3 replicas of tablet 02414b168fec4fa6aaec89e253356f6b
I20260619 20:27:44.813467 29780 tablet_service.cc:1514] Processing DeleteTablet for tablet 02414b168fec4fa6aaec89e253356f6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57600
I20260619 20:27:44.813496 30069 tablet_service.cc:1514] Processing DeleteTablet for tablet 02414b168fec4fa6aaec89e253356f6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:42642
I20260619 20:27:44.813599 30185 tablet_replica.cc:330] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9: stopping tablet replica
I20260619 20:27:44.813598 30186 tablet_replica.cc:330] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292: stopping tablet replica
I20260619 20:27:44.813585 29928 tablet_service.cc:1514] Processing DeleteTablet for tablet 02414b168fec4fa6aaec89e253356f6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:27:44 UTC) from {username='oryx', principal='oryx/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:54788
I20260619 20:27:44.813647 30185 raft_consensus.cc:2229] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:44.813652 30186 raft_consensus.cc:2229] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:27:44.813673 30184 tablet_replica.cc:330] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37: stopping tablet replica
I20260619 20:27:44.813689 30185 raft_consensus.cc:2258] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.813711 30186 raft_consensus.cc:2258] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.813730 30184 raft_consensus.cc:2229] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:27:44.813767 30184 raft_consensus.cc:2258] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:27:44.814021 30185 ts_tablet_manager.cc:1905] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.814090 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 29721
I20260619 20:27:44.814160 30186 ts_tablet_manager.cc:1905] T 02414b168fec4fa6aaec89e253356f6b P 76ec625d85604a7aa35181ce9c479292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:27:44.815759 30185 ts_tablet_manager.cc:1918] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:44.815824 30185 log.cc:1196] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/02414b168fec4fa6aaec89e253356f6b
I20260619 20:27:44.816037 30185 ts_tablet_manager.cc:1939] T 02414b168fec4fa6aaec89e253356f6b P 57d4cf0a03834a0e868042b90cf8b9a9: Deleting consensus metadata
I20260619 20:27:44.820044 29635 catalog_manager.cc:4841] TS 57d4cf0a03834a0e868042b90cf8b9a9 (127.21.151.3:42763): tablet 02414b168fec4fa6aaec89e253356f6b (table test-table [id=5e1c43b98acc498b93d9e2f26bcd5d77]) successfully deleted
I20260619 20:27:44.820372 30184 ts_tablet_manager.cc:1905] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:27:44.820375 29634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:37119 (error 108)
I20260619 20:27:44.820943 30245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.820743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37119 (local address 127.0.0.1:57634)
0619 20:27:44.820844 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.820894 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:37119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":63,"threads_started":1}
W20260619 20:27:44.821043 29634 catalog_manager.cc:4584] TS 76ec625d85604a7aa35181ce9c479292 (127.21.151.1:37119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02414b168fec4fa6aaec89e253356f6b: Network error: Client connection negotiation failed: client connection to 127.21.151.1:37119: connect: Connection refused (error 111)
I20260619 20:27:44.821985 30184 ts_tablet_manager.cc:1918] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 20:27:44.822049 30184 log.cc:1196] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/02414b168fec4fa6aaec89e253356f6b
I20260619 20:27:44.822264 30184 ts_tablet_manager.cc:1939] T 02414b168fec4fa6aaec89e253356f6b P 08126e498c2d4ae0b776ad4f138c2f37: Deleting consensus metadata
I20260619 20:27:44.822388 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 29862
I20260619 20:27:44.829010 30245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:44.828888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:37603 (local address 127.0.0.1:54796)
0619 20:27:44.828944 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:44.828980 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.2:37603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:27:44.829003 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 30003
I20260619 20:27:44.833921 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 29616
2026-06-19T20:27:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2756 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:44 dist-test-slave-f9rk krb5kdc[30248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:44 dist-test-slave-f9rk krb5kdc[30248](info): set up 2 sockets
Jun 19 20:27:44 dist-test-slave-f9rk krb5kdc[30248](info): commencing operation
krb5kdc: starting...
W20260619 20:27:45.869251 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:45 dist-test-slave-f9rk krb5kdc[30248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:45.982487 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:42819
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:42819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:46.060763 30261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:46.060949 30261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:46.060971 30261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:46.062033 30261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:46.062067 30261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:46.062080 30261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:46.062093 30261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:46.062104 30261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:46.063422 30261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:42819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:42819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:46.063584 30261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:46.063817 30261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:46.068465 30266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:46.068485 30269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:46.068493 30267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:46.068528 30261 server_base.cc:1034] running on GCE node
I20260619 20:27:46.068831 30261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:46.069106 30261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:46.070242 30261 hybrid_clock.cc:648] HybridClock initialized: now 1781900866070227 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T20:27:46Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1244 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:27:46 dist-test-slave-f9rk krb5kdc[30274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:27:46 dist-test-slave-f9rk krb5kdc[30274](info): set up 2 sockets
Jun 19 20:27:46 dist-test-slave-f9rk krb5kdc[30274](info): commencing operation
krb5kdc: starting...
W20260619 20:27:47.129379 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:27:47 dist-test-slave-f9rk krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:27:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:27:47.242019 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:45831
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:45831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:47.321928 30287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:47.322165 30287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:47.322209 30287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:47.323333 30287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:47.323396 30287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:47.323421 30287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:47.323452 30287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:47.323479 30287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:47.325021 30287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:45831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:45831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:47.325268 30287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:47.325491 30287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:47.330319 30293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:47.330356 30292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:47.330478 30287 server_base.cc:1034] running on GCE node
W20260619 20:27:47.330356 30295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:47.330747 30287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:47.330991 30287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:47.332127 30287 hybrid_clock.cc:648] HybridClock initialized: now 1781900867332102 us; error 42 us; skew 500 ppm
Jun 19 20:27:47 dist-test-slave-f9rk krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900867, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:47.338349 30287 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:47.338759 30287 webserver.cc:460] Webserver started at http://127.21.151.62:43675/ using document root <none> and password file <none>
I20260619 20:27:47.339023 30287 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:47.339093 30287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:47.339237 30287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:47.340350 30287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "b1341055a3c34db0961f2fff2149ce68"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "c2dfda5c23ca8a987cb909567696849a"
server_key_iv: "e309912f1597f661b92ff4c2f4e9ddbd"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.340744 30287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "b1341055a3c34db0961f2fff2149ce68"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "c2dfda5c23ca8a987cb909567696849a"
server_key_iv: "e309912f1597f661b92ff4c2f4e9ddbd"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.342021 30287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:47.342712 30302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:47.342881 30287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:47.342983 30287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "b1341055a3c34db0961f2fff2149ce68"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "c2dfda5c23ca8a987cb909567696849a"
server_key_iv: "e309912f1597f661b92ff4c2f4e9ddbd"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.343065 30287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:47.363794 30287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:47.365654 30287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:47.365823 30287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:47.369647 30287 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:45831
I20260619 20:27:47.369680 30355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:45831 every 8 connection(s)
I20260619 20:27:47.370093 30287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:47.370927 30356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:27:47.373088 30356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68: Bootstrap starting.
I20260619 20:27:47.373610 30356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68: Neither blocks nor log segments found. Creating new log.
I20260619 20:27:47.373834 30356 log.cc:824] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:47.374475 30356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68: No bootstrap required, opened a new log
I20260619 20:27:47.376044 30356 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } }
I20260619 20:27:47.376178 30356 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:27:47.376200 30356 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1341055a3c34db0961f2fff2149ce68, State: Initialized, Role: FOLLOWER
I20260619 20:27:47.376343 30356 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } }
I20260619 20:27:47.376420 30356 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:47.376448 30356 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:47.376482 30356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:27:47.376847 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 30287
I20260619 20:27:47.376966 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:27:47.377103 30356 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } }
I20260619 20:27:47.377164 22108 external_mini_cluster.cc:1428] Setting key e8f5f07609e0a0b25693237c5cbcaeb0
I20260619 20:27:47.377209 30356 leader_election.cc:302] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1341055a3c34db0961f2fff2149ce68; no voters:
I20260619 20:27:47.377375 30356 leader_election.cc:288] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:27:47.377409 30359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:27:47.377636 30359 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 1 LEADER]: Becoming Leader. State: Replica: b1341055a3c34db0961f2fff2149ce68, State: Running, Role: LEADER
I20260619 20:27:47.377801 30359 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } }
I20260619 20:27:47.377660 30356 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:47.378326 30360 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1341055a3c34db0961f2fff2149ce68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } } }
I20260619 20:27:47.378413 30360 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:47.378749 30362 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1341055a3c34db0961f2fff2149ce68. Latest consensus state: current_term: 1 leader_uuid: "b1341055a3c34db0961f2fff2149ce68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } } }
I20260619 20:27:47.378811 30362 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:47.379015 30367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:47.379617 30367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:47.381174 30367 catalog_manager.cc:1347] Generated new cluster ID: b2fb6994bbcd458f9f31305c0e62ff49
I20260619 20:27:47.381227 30367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:27:47 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:47.389729 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:47.377752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.21.151.62:45831)
0619 20:27:47.377894 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:47.377904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:47.377937 (+ 33us) server_negotiation.cc:407] Connection header received
0619 20:27:47.378061 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:47.378069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:47.378128 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:47.379149 (+ 1021us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:47.379347 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.380901 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.382146 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.382769 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.382880 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:47.385808 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:47.385812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:47.385812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:47.385826 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:47.388023 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:47.388414 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:47.388417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:47.388417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:47.388538 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:47.388784 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:47.388785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:47.388786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:47.389115 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:47.389266 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:47.389380 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 20:27:47.389514 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260619 20:27:47.403559 30367 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:27:47.404137 30367 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:47.414454 30367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68: Generated new TSK 0
I20260619 20:27:47.414680 30367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:27:47.440353 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45831
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:47.525902 30383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:47.526096 30383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:47.526120 30383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:47.527412 30383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:47.527456 30383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:47.527485 30383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:47.529099 30383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:47.529377 30383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:47.529644 30383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:47.534768 30389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:47.534818 30391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:47.534811 30383 server_base.cc:1034] running on GCE node
W20260619 20:27:47.534777 30388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:47.535220 30383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:47.535457 30383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:47.536631 30383 hybrid_clock.cc:648] HybridClock initialized: now 1781900867536610 us; error 41 us; skew 500 ppm
Jun 19 20:27:47 dist-test-slave-f9rk krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900867, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:47.539027 30383 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:27:47.539399 30383 webserver.cc:460] Webserver started at http://127.21.151.1:36475/ using document root <none> and password file <none>
I20260619 20:27:47.539678 30383 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:47.539726 30383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:47.539872 30383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:47.540884 30383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "c758f6e845d5456ba92dcde3a496b63e"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "937f0cfb79780f496edf05dc88e11fed"
server_key_iv: "3fece0a32b453fd4ffb4c5055cd26596"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.541205 30383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "c758f6e845d5456ba92dcde3a496b63e"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "937f0cfb79780f496edf05dc88e11fed"
server_key_iv: "3fece0a32b453fd4ffb4c5055cd26596"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.542503 30383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:27:47.543217 30398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:47.543510 30383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:47.543596 30383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "c758f6e845d5456ba92dcde3a496b63e"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "937f0cfb79780f496edf05dc88e11fed"
server_key_iv: "3fece0a32b453fd4ffb4c5055cd26596"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.543699 30383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:47.569811 30383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:47.572227 30383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:47.572341 30383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:47.572537 30383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:47.572790 30383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:47.572819 30383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:47.572840 30383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:47.572852 30383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:47.578470 30383 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:34559
I20260619 20:27:47.578532 30511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:34559 every 8 connection(s)
I20260619 20:27:47.578893 30383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:27:47 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900867, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:47.585155 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 30383
I20260619 20:27:47.585264 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:27:47.585455 22108 external_mini_cluster.cc:1428] Setting key b95526d15352256344f52ff6a2cb35c7
I20260619 20:27:47.588456 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:47.579541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:43655 (local address 127.21.151.62:45831)
0619 20:27:47.579604 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:47.579606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:47.579862 (+ 256us) server_negotiation.cc:407] Connection header received
0619 20:27:47.580404 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:47.580405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:47.580422 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:47.580467 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:47.581397 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.581936 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.582927 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.583141 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.583225 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:47.585390 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:47.585392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:47.585392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:47.585398 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:47.586886 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:47.587387 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:47.587388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:47.587388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:47.587466 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:47.587810 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:47.587811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:47.587811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:47.587928 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:47.588008 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:47.588280 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 20:27:47.588359 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:27:47.588644 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:47.579519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:43655)
0619 20:27:47.579726 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:47.579763 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:47.580289 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:47.580549 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:47.580554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:47.581118 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:47.581300 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:47.581305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.582040 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.582041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:47.582822 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:47.582825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.583237 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.583238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:47.583360 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:47.583834 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:47.583841 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:47.585271 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:47.586964 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:47.586971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:47.586973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:47.587315 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:47.587542 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:47.587543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:47.587544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:47.587735 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:47.587998 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:47.588004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:47.588144 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 20:27:47.588245 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 20:27:47.588398 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":131,"threads_started":1}
I20260619 20:27:47.589051 30512 heartbeater.cc:347] Connected to a master server at 127.21.151.62:45831
I20260619 20:27:47.589138 30512 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:47.589326 30512 heartbeater.cc:511] Master 127.21.151.62:45831 requested a full tablet report, sending...
I20260619 20:27:47.589811 30320 ts_manager.cc:194] Registered new tserver with Master: c758f6e845d5456ba92dcde3a496b63e (127.21.151.1:34559)
I20260619 20:27:47.590440 30320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:43655
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:27:47.607940 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45831
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:47.685978 30517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:47.686164 30517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:47.686185 30517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:47.687335 30517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:47.687373 30517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:47.687398 30517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:47.688827 30517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:47.689034 30517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:47.689230 30517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:47.693923 30525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:47.693943 30523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:47.693930 30522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:47.694031 30517 server_base.cc:1034] running on GCE node
I20260619 20:27:47.694320 30517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:47.694509 30517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:47.695688 30517 hybrid_clock.cc:648] HybridClock initialized: now 1781900867695649 us; error 56 us; skew 500 ppm
Jun 19 20:27:47 dist-test-slave-f9rk krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900867, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:47.698175 30517 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:27:47.698637 30517 webserver.cc:460] Webserver started at http://127.21.151.2:42679/ using document root <none> and password file <none>
I20260619 20:27:47.698872 30517 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:47.698916 30517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:47.699054 30517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:47.700104 30517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "eee41fae92524248a5c41098aae2d70e"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "b3a491dd48e2e2342056d11c922c860c"
server_key_iv: "53e42b306d7257febb47f2b2d77b2ff3"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.700435 30517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "eee41fae92524248a5c41098aae2d70e"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "b3a491dd48e2e2342056d11c922c860c"
server_key_iv: "53e42b306d7257febb47f2b2d77b2ff3"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.701717 30517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:47.702440 30532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:47.702642 30517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:47.702710 30517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "eee41fae92524248a5c41098aae2d70e"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "b3a491dd48e2e2342056d11c922c860c"
server_key_iv: "53e42b306d7257febb47f2b2d77b2ff3"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.702798 30517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:47.713721 30517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:47.715950 30517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:47.716143 30517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:47.716364 30517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:47.716663 30517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:47.716717 30517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:47.716749 30517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:47.716777 30517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:47.722424 30517 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:39063
I20260619 20:27:47.722471 30646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:39063 every 8 connection(s)
I20260619 20:27:47.722767 30517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:27:47 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900867, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:47.731256 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:47.723439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:50815 (local address 127.21.151.62:45831)
0619 20:27:47.723482 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:47.723485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:47.723719 (+ 234us) server_negotiation.cc:407] Connection header received
0619 20:27:47.724182 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:47.724183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:47.724192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:47.724253 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:47.725123 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.725621 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.726540 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.726748 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.726797 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:47.728909 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:47.728910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:47.728911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:47.728917 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:47.730049 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:47.730386 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:47.730387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:47.730387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:47.730434 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:47.730644 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:47.730644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:47.730645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:47.730740 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:47.730788 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:47.731068 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 20:27:47.731153 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:27:47.731407 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:47.723392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:50815)
0619 20:27:47.723546 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:47.723587 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:47.724098 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:47.724305 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:47.724309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:47.724859 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:47.725045 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:47.725050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.725703 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.725704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:47.726465 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:47.726466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.726824 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.726825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:47.726955 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:47.727446 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:47.727454 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:47.728820 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:47.730116 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:47.730121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:47.730123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:47.730324 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:47.730500 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:47.730501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:47.730501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:47.730592 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:47.730792 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:47.730796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:47.730897 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 20:27:47.731036 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 20:27:47.731182 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260619 20:27:47.731979 30647 heartbeater.cc:347] Connected to a master server at 127.21.151.62:45831
I20260619 20:27:47.732060 30647 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:47.732189 30647 heartbeater.cc:511] Master 127.21.151.62:45831 requested a full tablet report, sending...
I20260619 20:27:47.732324 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 30517
I20260619 20:27:47.732450 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:27:47.732575 30320 ts_manager.cc:194] Registered new tserver with Master: eee41fae92524248a5c41098aae2d70e (127.21.151.2:39063)
I20260619 20:27:47.732698 22108 external_mini_cluster.cc:1428] Setting key 998ebbf762c8c81e0a7cfb36b806ac26
I20260619 20:27:47.732995 30320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:50815
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:27:47.755057 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45831
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:47.833673 30652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:47.833921 30652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:47.833954 30652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:47.835105 30652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:47.835186 30652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:47.835260 30652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:47.836658 30652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:45831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:47.836912 30652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:47.837132 30652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:47.841980 30657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:47.842006 30660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:47.842233 30652 server_base.cc:1034] running on GCE node
W20260619 20:27:47.841993 30658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:47.842501 30652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:47.842684 30652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:47.843816 30652 hybrid_clock.cc:648] HybridClock initialized: now 1781900867843805 us; error 28 us; skew 500 ppm
Jun 19 20:27:47 dist-test-slave-f9rk krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900867, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:47.846396 30652 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:27:47.846912 30652 webserver.cc:460] Webserver started at http://127.21.151.3:37897/ using document root <none> and password file <none>
I20260619 20:27:47.847136 30652 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:47.847180 30652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:47.847378 30652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:27:47.848445 30652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "4da33b8471c54f1d86bddaaad0440848"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "3cd6488ae066caaf863843d33bad660f"
server_key_iv: "1a752e46350356a8cbb339fd1f6159fe"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.848834 30652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "4da33b8471c54f1d86bddaaad0440848"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "3cd6488ae066caaf863843d33bad660f"
server_key_iv: "1a752e46350356a8cbb339fd1f6159fe"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.850081 30652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:47.850847 30667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:47.851045 30652 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:47.851140 30652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "4da33b8471c54f1d86bddaaad0440848"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "3cd6488ae066caaf863843d33bad660f"
server_key_iv: "1a752e46350356a8cbb339fd1f6159fe"
server_key_version: "encryptionkey@0"
I20260619 20:27:47.851228 30652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:47.867738 30652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:47.870108 30652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:47.870270 30652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:47.870504 30652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:47.870807 30652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:47.870859 30652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:47.870893 30652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:47.870921 30652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:47.876474 30652 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:46201
I20260619 20:27:47.876511 30781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:46201 every 8 connection(s)
I20260619 20:27:47.876837 30652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:47.879611 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 30652
I20260619 20:27:47.879789 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:27:47.880002 22108 external_mini_cluster.cc:1428] Setting key 16fc62a0ca4ce085ac1269f911874c25
Jun 19 20:27:47 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900867, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:27:47.885849 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:47.877501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:37163 (local address 127.21.151.62:45831)
0619 20:27:47.877541 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:47.877543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:47.877743 (+ 200us) server_negotiation.cc:407] Connection header received
0619 20:27:47.878362 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:47.878364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:47.878377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:47.878413 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:47.879478 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.880268 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.881249 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.881459 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.881550 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:47.883574 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:47.883575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:47.883576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:47.883582 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:47.884762 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:47.885112 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:47.885113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:47.885113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:47.885162 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:47.885364 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:47.885365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:47.885365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:47.885459 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:47.885507 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:47.885729 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 20:27:47.885788 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:27:47.886039 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:47.877493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:37163)
0619 20:27:47.877628 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:47.877661 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:47.878250 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:47.878495 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:47.878499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:47.879171 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:47.879350 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:47.879355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.880339 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.880340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:47.881144 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:47.881146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:47.881558 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:47.881559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:47.881683 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:47.882198 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:47.882206 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:47.883483 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:27:47.884829 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:47.884834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:47.884835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:47.885047 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:47.885220 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:27:47.885220 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:27:47.885220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:27:47.885312 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:27:47.885516 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:27:47.885520 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:27:47.885624 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 20:27:47.885702 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:27:47.885810 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260619 20:27:47.886521 30782 heartbeater.cc:347] Connected to a master server at 127.21.151.62:45831
I20260619 20:27:47.886596 30782 heartbeater.cc:464] Registering TS with master...
I20260619 20:27:47.886793 30782 heartbeater.cc:511] Master 127.21.151.62:45831 requested a full tablet report, sending...
I20260619 20:27:47.887158 30319 ts_manager.cc:194] Registered new tserver with Master: 4da33b8471c54f1d86bddaaad0440848 (127.21.151.3:46201)
I20260619 20:27:47.887494 30319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:37163
I20260619 20:27:47.891568 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab.
Entry for principal oryx/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab.
WARNING: no policy specified for oryx/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab.
Entry for principal oryx/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab.
WARNING: no policy specified for oryx/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab.
Entry for principal oryx/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab.
I20260619 20:27:47.954641 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 30383
I20260619 20:27:47.959133 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 30517
I20260619 20:27:47.963260 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 30652
I20260619 20:27:47.967451 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 30287
I20260619 20:27:47.972412 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:45831
--webserver_interface=127.21.151.62
--webserver_port=43675
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:45831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:27:48.051249 30791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:48.051501 30791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:48.051540 30791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:48.052733 30791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:27:48.052809 30791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:48.052834 30791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:48.052853 30791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:27:48.052884 30791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:27:48.054353 30791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:45831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:45831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=43675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:48.054625 30791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:48.054842 30791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:48.059396 30797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:48.059564 30791 server_base.cc:1034] running on GCE node
W20260619 20:27:48.059403 30796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:48.059669 30799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:48.059958 30791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:48.060190 30791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:48.061327 30791 hybrid_clock.cc:648] HybridClock initialized: now 1781900868061315 us; error 26 us; skew 500 ppm
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900868, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:48.063845 30791 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:27:48.064257 30791 webserver.cc:460] Webserver started at http://127.21.151.62:43675/ using document root <none> and password file <none>
I20260619 20:27:48.064505 30791 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:48.064580 30791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:48.065887 30791 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:48.066557 30806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:48.066730 30791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:48.066824 30791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "b1341055a3c34db0961f2fff2149ce68"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "c2dfda5c23ca8a987cb909567696849a"
server_key_iv: "e309912f1597f661b92ff4c2f4e9ddbd"
server_key_version: "encryptionkey@0"
I20260619 20:27:48.067181 30791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:48.090790 30791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:48.092702 30791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:48.092868 30791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:48.096616 30791 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:45831
I20260619 20:27:48.096704 30859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:45831 every 8 connection(s)
I20260619 20:27:48.097060 30791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:27:48.098465 30860 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:27:48.099035 30860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68: Bootstrap starting.
I20260619 20:27:48.107017 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 30791
I20260619 20:27:48.107100 30860 log.cc:824] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68: Log is configured to *not* fsync() on all Append() calls
I20260619 20:27:48.107388 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:34559
--local_ip_for_outbound_sockets=127.21.151.1
--tserver_master_addrs=127.21.151.62:45831
--webserver_port=36475
--webserver_interface=127.21.151.1
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.151.1
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:27:48.109244 30860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:27:48.109549 30860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68: Bootstrap complete.
I20260619 20:27:48.112082 30860 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } }
I20260619 20:27:48.112300 30860 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1341055a3c34db0961f2fff2149ce68, State: Initialized, Role: FOLLOWER
I20260619 20:27:48.112461 30860 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } }
I20260619 20:27:48.112552 30860 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:27:48.112609 30860 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:27:48.112661 30860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:27:48.113658 30860 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } }
I20260619 20:27:48.113770 30860 leader_election.cc:302] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1341055a3c34db0961f2fff2149ce68; no voters:
I20260619 20:27:48.113973 30860 leader_election.cc:288] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:27:48.114012 30864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:27:48.114202 30864 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [term 2 LEADER]: Becoming Leader. State: Replica: b1341055a3c34db0961f2fff2149ce68, State: Running, Role: LEADER
I20260619 20:27:48.114282 30860 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:27:48.114442 30864 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } }
I20260619 20:27:48.115545 30866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1341055a3c34db0961f2fff2149ce68. Latest consensus state: current_term: 2 leader_uuid: "b1341055a3c34db0961f2fff2149ce68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } } }
I20260619 20:27:48.115542 30865 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b1341055a3c34db0961f2fff2149ce68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1341055a3c34db0961f2fff2149ce68" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 45831 } } }
I20260619 20:27:48.115705 30866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:48.115710 30865 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68 [sys.catalog]: This master's current role is: LEADER
I20260619 20:27:48.116078 30879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:27:48.116598 30879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:27:48.116709 30879 catalog_manager.cc:1259] Loaded cluster ID: b2fb6994bbcd458f9f31305c0e62ff49
I20260619 20:27:48.116734 30879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:27:48.117834 30879 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:27:48.117987 30879 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b1341055a3c34db0961f2fff2149ce68: Loaded TSK: 0
I20260619 20:27:48.118188 30879 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:27:48.202064 30863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:48.202359 30863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:48.202414 30863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:48.204677 30863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:48.204736 30863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:48.204785 30863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:27:48.207404 30863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.1.keytab
--principal=oryx/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:34559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=36475
--tserver_master_addrs=127.21.151.62:45831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:48.207701 30863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:48.207984 30863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:48.213469 30887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:48.213466 30889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:48.214164 30886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:48.214725 30863 server_base.cc:1034] running on GCE node
I20260619 20:27:48.214964 30863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:48.215270 30863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:48.216445 30863 hybrid_clock.cc:648] HybridClock initialized: now 1781900868216420 us; error 37 us; skew 500 ppm
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900868, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:48.219300 30863 init.cc:377] Logged in from keytab as oryx/127.21.151.1@KRBTEST.COM (short username oryx)
I20260619 20:27:48.219802 30863 webserver.cc:460] Webserver started at http://127.21.151.1:36475/ using document root <none> and password file <none>
I20260619 20:27:48.220045 30863 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:48.220124 30863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:48.221552 30863 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:27:48.222174 30896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:48.222352 30863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:27:48.222452 30863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "c758f6e845d5456ba92dcde3a496b63e"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "937f0cfb79780f496edf05dc88e11fed"
server_key_iv: "3fece0a32b453fd4ffb4c5055cd26596"
server_key_version: "encryptionkey@0"
I20260619 20:27:48.222800 30863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:48.234856 30863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:48.237177 30863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:48.237334 30863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:48.237581 30863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:48.237924 30863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:48.237984 30863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:48.238025 30863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:48.238054 30863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:48.245223 30863 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:34559
I20260619 20:27:48.245268 31010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:34559 every 8 connection(s)
I20260619 20:27:48.245596 30863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:27:48.253293 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 30863
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:48.253628 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:39063
--local_ip_for_outbound_sockets=127.21.151.2
--tserver_master_addrs=127.21.151.62:45831
--webserver_port=42679
--webserver_interface=127.21.151.2
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.151.2
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:48.255267 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:48.246266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:55297)
0619 20:27:48.246433 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:48.246477 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:48.247082 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:48.247741 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:48.247744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:48.248283 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:48.248478 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.248482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.250103 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.250104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.250867 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.250869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.251746 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.251747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.251845 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.252415 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:48.252423 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:48.255124 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
W20260619 20:27:48.255528 31011 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:45831 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.151.62:45831: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
W20260619 20:27:48.255990 31014 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:48.246341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:55297 (local address 127.21.151.62:45831)
0619 20:27:48.247265 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:48.247273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:48.247307 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:27:48.247436 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:48.247441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:48.247502 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:48.247650 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:48.248597 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.250007 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.250960 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.251664 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.251810 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.255785 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:55297: BlockingRecv error: recv got EOF from 127.21.151.1:55297 (error 108)
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":78,"threads_started":1}
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:48.260357 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:48.255760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:38053)
0619 20:27:48.256026 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:48.256039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:48.256087 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:48.256373 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:48.256374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:48.256496 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:48.256536 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.256538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.257183 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.257184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.257749 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.257751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.258101 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.258102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.258171 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.258617 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:48.258622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:48.260277 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":63,"threads_started":1}
W20260619 20:27:48.260416 31017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:48.255905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:38053 (local address 127.21.151.62:45831)
0619 20:27:48.256205 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:48.256207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:48.256214 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:27:48.256236 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:48.256236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:48.256247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:48.256339 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:48.256614 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.257093 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.257831 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.258027 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.258104 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.260351 (+ 2247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:38053: BlockingRecv error: recv got EOF from 127.21.151.1:38053 (error 108)
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":69,"threads_started":1}
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:48.264807 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:48.260664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:42259)
0619 20:27:48.260833 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:48.260843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:48.260911 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:48.261042 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:48.261044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:48.261174 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:48.261198 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.261200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.261797 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.261797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.262285 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.262287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.262628 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.262629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.262678 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.263089 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:48.263094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:48.264730 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156}
W20260619 20:27:48.264858 31017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:48.260755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:42259 (local address 127.21.151.62:45831)
0619 20:27:48.260814 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:48.260816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:48.260898 (+ 82us) server_negotiation.cc:407] Connection header received
0619 20:27:48.260963 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:48.260964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:48.260973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:48.261040 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:48.261272 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.261720 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.262367 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.262558 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.262630 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.264799 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:42259: BlockingRecv error: recv got EOF from 127.21.151.1:42259 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 20:27:48.265144 31011 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:27:48.336485 31015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:48.336731 31015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:48.336755 31015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:48.337827 31015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:48.337862 31015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:48.337885 31015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:27:48.339176 31015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.2.keytab
--principal=oryx/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:39063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=42679
--tserver_master_addrs=127.21.151.62:45831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:48.339358 31015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:48.339526 31015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:48.344415 31022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:48.344415 31023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:48.344556 31025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:48.344821 31015 server_base.cc:1034] running on GCE node
I20260619 20:27:48.345065 31015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:48.345289 31015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:48.346493 31015 hybrid_clock.cc:648] HybridClock initialized: now 1781900868346476 us; error 26 us; skew 500 ppm
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900868, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:48.349257 31015 init.cc:377] Logged in from keytab as oryx/127.21.151.2@KRBTEST.COM (short username oryx)
I20260619 20:27:48.349766 31015 webserver.cc:460] Webserver started at http://127.21.151.2:42679/ using document root <none> and password file <none>
I20260619 20:27:48.350061 31015 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:48.350155 31015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:48.351718 31015 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:27:48.352396 31032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:48.352566 31015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:48.352680 31015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "eee41fae92524248a5c41098aae2d70e"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "b3a491dd48e2e2342056d11c922c860c"
server_key_iv: "53e42b306d7257febb47f2b2d77b2ff3"
server_key_version: "encryptionkey@0"
I20260619 20:27:48.353066 31015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:48.394656 31015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:48.396988 31015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:48.397161 31015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:48.397401 31015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:48.397715 31015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:48.397766 31015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:48.397810 31015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:48.397835 31015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:48.403542 31015 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:39063
I20260619 20:27:48.403662 31146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:39063 every 8 connection(s)
I20260619 20:27:48.403983 31015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:27:48.409338 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 31015
I20260619 20:27:48.409628 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:46201
--local_ip_for_outbound_sockets=127.21.151.3
--tserver_master_addrs=127.21.151.62:45831
--webserver_port=37897
--webserver_interface=127.21.151.3
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.151.3
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:48.411335 31149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:48.404745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:39581)
0619 20:27:48.404903 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:48.404939 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:48.405463 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:48.405673 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:48.405676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:48.406232 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:48.406490 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.406497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.407155 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.407155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.407971 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.407973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.408298 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.408299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.408394 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.408849 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:48.408857 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:48.411195 (+ 2338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
W20260619 20:27:48.411387 31017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:48.404738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:39581 (local address 127.21.151.62:45831)
0619 20:27:48.404786 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:48.404788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:48.405051 (+ 263us) server_negotiation.cc:407] Connection header received
0619 20:27:48.405550 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:48.405551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:48.405564 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:48.405613 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:48.406588 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.407076 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.408045 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.408228 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.408280 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.411312 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:39581: BlockingRecv error: recv got EOF from 127.21.151.2:39581 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:27:48.411715 31147 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:45831 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.151.62:45831: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:48.416937 31149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:48.411951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:35641)
0619 20:27:48.412005 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:48.412018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:48.412065 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:48.412216 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:48.412216 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:48.412323 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:48.412364 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.412365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.413262 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.413263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.413811 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.413812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.414393 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.414394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.414439 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.414957 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:48.414961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:48.416894 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 20:27:48.417097 31017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:48.412027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:35641 (local address 127.21.151.62:45831)
0619 20:27:48.412088 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:48.412090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:48.412099 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:48.412138 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:48.412139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:48.412151 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:48.412206 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:48.412432 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.413189 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.413897 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.414272 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.414384 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.417019 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:35641: BlockingRecv error: recv got EOF from 127.21.151.2:35641 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:48.421298 31149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:48.417361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:46447)
0619 20:27:48.417430 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:48.417440 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:48.417499 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:48.417616 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:48.417617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:48.417737 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:48.417761 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.417762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.418374 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.418375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.418834 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.418835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.419185 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.419185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.419234 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.419688 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:48.419696 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:48.421225 (+ 1529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 20:27:48.421347 31017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:48.417455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:46447 (local address 127.21.151.62:45831)
0619 20:27:48.417520 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:48.417522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:48.417530 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:48.417553 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:48.417554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:48.417562 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:48.417605 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:48.417832 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.418292 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.418918 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.419108 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.419174 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.421290 (+ 2116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:46447: BlockingRecv error: recv got EOF from 127.21.151.2:46447 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 20:27:48.421622 31147 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:27:48.492053 31150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:27:48.492245 31150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:27:48.492268 31150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:27:48.493448 31150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:27:48.493489 31150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:27:48.493515 31150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:27:48.495012 31150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/krb5kdc/oryx_127.21.151.3.keytab
--principal=oryx/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:46201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=37897
--tserver_master_addrs=127.21.151.62:45831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:27:48.495263 31150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:27:48.495496 31150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:27:48.500389 31156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:27:48.500422 31155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:48.500612 31150 server_base.cc:1034] running on GCE node
W20260619 20:27:48.500641 31158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:27:48.500857 31150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:27:48.501051 31150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:27:48.502187 31150 hybrid_clock.cc:648] HybridClock initialized: now 1781900868502169 us; error 31 us; skew 500 ppm
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900868, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:27:48.504726 31150 init.cc:377] Logged in from keytab as oryx/127.21.151.3@KRBTEST.COM (short username oryx)
I20260619 20:27:48.505098 31150 webserver.cc:460] Webserver started at http://127.21.151.3:37897/ using document root <none> and password file <none>
I20260619 20:27:48.505290 31150 fs_manager.cc:359] Metadata directory not provided
I20260619 20:27:48.505338 31150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:27:48.506610 31150 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:27:48.507212 31165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:48.507381 31150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:27:48.507476 31150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "4da33b8471c54f1d86bddaaad0440848"
format_stamp: "Formatted at 2026-06-19 20:27:47 on dist-test-slave-f9rk"
server_key: "3cd6488ae066caaf863843d33bad660f"
server_key_iv: "1a752e46350356a8cbb339fd1f6159fe"
server_key_version: "encryptionkey@0"
I20260619 20:27:48.507884 31150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:27:48.521817 31150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:27:48.523584 31150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:27:48.523769 31150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:27:48.523984 31150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:27:48.524300 31150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:27:48.524358 31150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:48.524400 31150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:27:48.524427 31150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:27:48.530043 31150 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:46201
I20260619 20:27:48.530120 31279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:46201 every 8 connection(s)
I20260619 20:27:48.530438 31150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:27:48.534965 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 31150
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 20:27:48.538336 31017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:48.531127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:43469 (local address 127.21.151.62:45831)
0619 20:27:48.531188 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:48.531189 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:48.531431 (+ 242us) server_negotiation.cc:407] Connection header received
0619 20:27:48.531928 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:48.531929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:48.531939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:48.531995 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:48.532855 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.533344 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.534608 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.534833 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.534902 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.538280 (+ 3378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:43469: BlockingRecv error: recv got EOF from 127.21.151.3:43469 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:27:48.538493 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:48.531098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:43469)
0619 20:27:48.531277 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:48.531325 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:48.531841 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:48.532054 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:48.532058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:48.532600 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:48.532783 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.532788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.533418 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.533419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.534512 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.534516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.534924 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.534924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.535080 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.535740 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:48.535750 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:48.538221 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
W20260619 20:27:48.538781 31280 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:45831 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.151.62:45831: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:48.542981 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:48.535508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.21.151.62:45831)
0619 20:27:48.535564 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:48.535566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:48.535576 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:48.535667 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:48.535667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:48.535683 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:48.535752 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:48.536349 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.536988 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.538050 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.538222 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.538341 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.539781 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:27:48.539785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:27:48.539785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:27:48.539796 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:27:48.541539 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:48.541925 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:48.541928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:48.541929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:48.542025 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:27:48.542247 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:27:48.542248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:27:48.542248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:27:48.542479 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:27:48.542624 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 20:27:48.542779 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 20:27:48.542892 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 20:27:48.543185 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:48.538982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:56865)
0619 20:27:48.539122 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:48.539131 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:48.539174 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:48.539324 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:48.539324 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:48.539455 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:48.539498 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.539499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.540139 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.540140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.540639 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.540641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.541060 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.541061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.541132 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.541495 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:48.541499 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:48.543128 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
W20260619 20:27:48.543265 31017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:48.539055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:56865 (local address 127.21.151.62:45831)
0619 20:27:48.539097 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:48.539099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:48.539162 (+ 63us) server_negotiation.cc:407] Connection header received
0619 20:27:48.539230 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:48.539231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:48.539240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:48.539306 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:48.539580 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.540059 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.540736 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.540999 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.541062 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.543212 (+ 2150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:56865: BlockingRecv error: recv got EOF from 127.21.151.3:56865 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:48 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:48.547302 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:48.543651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:47319)
0619 20:27:48.543737 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:48.543748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:48.543799 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:48.543945 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:48.543946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:48.544053 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:48.544076 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.544077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.544666 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.544667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.545142 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:48.545143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.545495 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.545496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:48.545548 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.545862 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:48.545866 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:48.547262 (+ 1396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260619 20:27:48.547420 31017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:48.543706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:47319 (local address 127.21.151.62:45831)
0619 20:27:48.543738 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:48.543740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:48.543793 (+ 53us) server_negotiation.cc:407] Connection header received
0619 20:27:48.543857 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:48.543858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:48.543866 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:48.543891 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:48.544156 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.544590 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.545235 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:48.545414 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:48.545494 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:48.547352 (+ 1858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:47319: BlockingRecv error: recv got EOF from 127.21.151.3:47319 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 20:27:48.547420 31280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 20:27:49 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:49 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:49.270515 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:49.265609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:45853)
0619 20:27:49.265809 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:49.265822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:49.265881 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:49.266102 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:49.266102 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:49.266259 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:49.266308 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:49.266309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.266983 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.266983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:49.267525 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:49.267526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.267904 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.267905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:49.267949 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:49.268408 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:49.268412 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:49.270439 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":65,"threads_started":1}
W20260619 20:27:49.270584 31286 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:49.265820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:45853 (local address 127.21.151.62:45831)
0619 20:27:49.265949 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:49.265951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:49.265961 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:49.266000 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:49.266001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:49.266012 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:49.266080 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:49.266378 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.266911 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.267607 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.267812 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.267895 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:49.270533 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:45853: BlockingRecv error: recv got EOF from 127.21.151.1:45853 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
Jun 19 20:27:49 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:49 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:49.426378 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:49.422035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:53395)
0619 20:27:49.422175 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:49.422185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:49.422237 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:49.422335 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:49.422336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:49.422492 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:49.422546 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:49.422548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.423147 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.423148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:49.423755 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:49.423757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.424067 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.424068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:49.424108 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:49.424543 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:49.424547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:49.426314 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260619 20:27:49.426477 31286 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:49.422080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:53395 (local address 127.21.151.62:45831)
0619 20:27:49.422151 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:49.422153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:49.422248 (+ 95us) server_negotiation.cc:407] Connection header received
0619 20:27:49.422273 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:49.422274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:49.422283 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:49.422332 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:49.422602 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.423083 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.423823 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.424005 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.424055 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:49.426404 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:53395: BlockingRecv error: recv got EOF from 127.21.151.2:53395 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 20:27:49 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:49 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:49.552348 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:49.547774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:55129)
0619 20:27:49.547907 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:49.547918 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:49.547972 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:49.548117 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:49.548117 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:49.548272 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:49.548319 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:49.548320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.548980 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.548981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:49.549710 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:49.549711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.550047 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.550048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:49.550087 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:49.550519 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:49.550523 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:49.552283 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
W20260619 20:27:49.552415 31286 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:49.547850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:55129 (local address 127.21.151.62:45831)
0619 20:27:49.547905 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:49.547907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:49.547969 (+ 62us) server_negotiation.cc:407] Connection header received
0619 20:27:49.548029 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:49.548030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:49.548046 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:49.548097 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:49.548378 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.548879 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.549774 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:49.549980 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:49.550040 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:49.552369 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:55129: BlockingRecv error: recv got EOF from 127.21.151.3:55129 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 20:27:50 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:50 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:50.275548 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:50.270992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:46913)
0619 20:27:50.271125 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:50.271136 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:50.271193 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:50.271327 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:50.271328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:50.271483 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:50.271534 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:50.271535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.272165 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.272166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:50.272678 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:50.272679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.272988 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.272989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:50.273036 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:50.273510 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:50.273515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:50.275490 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
W20260619 20:27:50.275692 31290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:50.271056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:46913 (local address 127.21.151.62:45831)
0619 20:27:50.271200 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:50.271202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:50.271211 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:50.271236 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:50.271236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:50.271246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:50.271309 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:50.271598 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.272103 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.272746 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.272927 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.272976 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:50.275580 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:46913: BlockingRecv error: recv got EOF from 127.21.151.1:46913 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
Jun 19 20:27:50 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:50 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:50.431452 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:50.426828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:57181)
0619 20:27:50.426991 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:50.427002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:50.427090 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:50.427276 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:50.427278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:50.427460 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:50.427512 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:50.427514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.428243 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.428244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:50.428789 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:50.428790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.429168 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.429169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:50.429229 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:50.429657 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:50.429662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:50.431359 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
W20260619 20:27:50.431500 31290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:50.426883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:57181 (local address 127.21.151.62:45831)
0619 20:27:50.426950 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:50.426952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:50.427078 (+ 126us) server_negotiation.cc:407] Connection header received
0619 20:27:50.427183 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:50.427184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:50.427197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:50.427269 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:50.427629 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.428146 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.428877 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.429082 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.429168 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:50.431435 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:57181: BlockingRecv error: recv got EOF from 127.21.151.2:57181 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 20:27:50 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:50 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:50.557575 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:50.552858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:44423)
0619 20:27:50.552982 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:50.552993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:50.553087 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:50.553249 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:50.553250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:50.553432 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:50.553478 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:50.553479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.554193 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.554194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:50.554721 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:50.554723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.555113 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.555115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:50.555182 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:50.555784 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:50.555789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:50.557517 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
W20260619 20:27:50.557703 31290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:50.552992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:44423 (local address 127.21.151.62:45831)
0619 20:27:50.553069 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:50.553071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:50.553082 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:27:50.553156 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:50.553157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:50.553173 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:50.553228 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:50.553564 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.554104 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.554817 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:50.555007 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:50.555112 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:50.557603 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:44423: BlockingRecv error: recv got EOF from 127.21.151.3:44423 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 20:27:51 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:51 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:51.280429 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:51.276077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:38265)
0619 20:27:51.276217 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:51.276227 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:51.276275 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:51.276422 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:51.276422 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:51.276593 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:51.276642 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:51.276644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.277265 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.277266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:51.277797 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:51.277799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.278122 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.278123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:51.278163 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:51.278611 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:51.278615 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:51.280369 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
W20260619 20:27:51.280591 31294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:51.276167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:38265 (local address 127.21.151.62:45831)
0619 20:27:51.276298 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:51.276300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:51.276308 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:51.276330 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:51.276331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:51.276341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:51.276403 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:51.276710 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.277202 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.277869 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.278059 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.278109 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:51.280464 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:38265: BlockingRecv error: recv got EOF from 127.21.151.1:38265 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
Jun 19 20:27:51 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:51 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:51.436926 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:51.432017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:45315)
0619 20:27:51.432156 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:51.432170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:51.432236 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:51.432404 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:51.432405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:51.432597 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:51.432641 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:51.432643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.433322 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.433323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:51.433857 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:51.433858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.434273 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.434274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:51.434330 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:51.434913 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:51.434920 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:51.436868 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
W20260619 20:27:51.437017 31294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:51.432136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:45315 (local address 127.21.151.62:45831)
0619 20:27:51.432210 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:51.432212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:51.432223 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:27:51.432305 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:51.432306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:51.432319 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:51.432395 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:51.432725 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.433232 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.433954 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.434160 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.434244 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:51.436961 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:45315: BlockingRecv error: recv got EOF from 127.21.151.2:45315 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 20:27:51 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:51 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:51.562304 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:51.558010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:57419)
0619 20:27:51.558151 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:51.558161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:51.558215 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:51.558346 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:51.558347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:51.558516 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:51.558564 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:51.558566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.559198 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.559198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:51.559732 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:51.559733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.560077 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.560078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:51.560118 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:51.560565 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:51.560569 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:51.562231 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
W20260619 20:27:51.562404 31294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:51.558089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:57419 (local address 127.21.151.62:45831)
0619 20:27:51.558161 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:51.558163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:51.558206 (+ 43us) server_negotiation.cc:407] Connection header received
0619 20:27:51.558261 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:51.558261 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:51.558274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:51.558327 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:51.558634 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.559132 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.559807 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:51.560010 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:51.560065 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:51.562326 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:57419: BlockingRecv error: recv got EOF from 127.21.151.3:57419 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 20:27:52 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:52 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:52.286069 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:52.280988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:46785)
0619 20:27:52.281137 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:52.281150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:52.281235 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:52.281483 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:52.281484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:52.281703 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:52.281755 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:52.281757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.282509 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.282509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:52.283099 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:52.283100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.283493 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.283494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:52.283537 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:52.284052 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:52.284057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:52.286006 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
W20260619 20:27:52.286248 31298 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:52.281029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:46785 (local address 127.21.151.62:45831)
0619 20:27:52.281203 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:52.281206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:52.281219 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:27:52.281341 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:52.281342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:52.281358 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:52.281454 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:52.281859 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.282422 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.283189 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.283416 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.283484 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:52.286140 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:46785: BlockingRecv error: recv got EOF from 127.21.151.1:46785 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
Jun 19 20:27:52 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:52 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:52.446081 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:52.437540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:50341)
0619 20:27:52.437730 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:52.437743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:52.437837 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:52.438016 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:52.438017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:52.438200 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:52.438259 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:52.438261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.439017 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.439018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:52.439560 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:52.439561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.440020 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.440021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:52.440082 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:52.444242 (+ 4160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:52.444246 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:52.446016 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":110,"threads_started":1}
W20260619 20:27:52.446233 31298 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:52.437637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:50341 (local address 127.21.151.62:45831)
0619 20:27:52.437717 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:52.437720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:52.437819 (+ 99us) server_negotiation.cc:407] Connection header received
0619 20:27:52.437895 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:52.437896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:52.437909 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:52.438004 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:52.438378 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.438872 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.439708 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.439907 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.440021 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:52.446148 (+ 6127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:50341: BlockingRecv error: recv got EOF from 127.21.151.2:50341 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 20:27:52 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:52 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:52.567422 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:52.562850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:43103)
0619 20:27:52.563032 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:52.563062 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:52.563139 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:52.563259 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:52.563259 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:52.563415 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:52.563464 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:52.563466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.564094 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.564095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:52.564617 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:52.564619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.564930 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.564931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:52.564970 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:52.565517 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:52.565522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:52.567325 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":65,"threads_started":1}
W20260619 20:27:52.567464 31298 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:52.562898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:43103 (local address 127.21.151.62:45831)
0619 20:27:52.562945 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:52.562949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:52.563138 (+ 189us) server_negotiation.cc:407] Connection header received
0619 20:27:52.563182 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:52.563182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:52.563198 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:52.563245 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:52.563524 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.564030 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.564687 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:52.564867 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:52.564916 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:52.567403 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:43103: BlockingRecv error: recv got EOF from 127.21.151.3:43103 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 20:27:53 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:53 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:53.291028 31301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:53.286628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:40257)
0619 20:27:53.286776 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:53.286786 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:53.286833 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:53.286962 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:53.286963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:53.287117 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:53.287164 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:53.287166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.287792 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.287793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:53.288298 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:53.288300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.288612 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.288613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:53.288657 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:53.289132 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:53.289136 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:53.290937 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260619 20:27:53.291100 31302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:53.286704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:40257 (local address 127.21.151.62:45831)
0619 20:27:53.286839 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:53.286840 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:53.286851 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:27:53.286874 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:53.286874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:53.286883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:53.286944 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:53.287230 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.287729 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.288371 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.288550 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.288597 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:53.291030 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:40257: BlockingRecv error: recv got EOF from 127.21.151.1:40257 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
Jun 19 20:27:53 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:53 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:53.451202 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:53.446630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:47995)
0619 20:27:53.446790 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:53.446802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:53.446863 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:53.447023 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:53.447024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:53.447228 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:53.447278 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:53.447280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.447994 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.447995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:53.448536 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:53.448539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.448863 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.448864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:53.448905 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:53.449359 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:53.449364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:53.451144 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
W20260619 20:27:53.451385 31302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:53.446763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:47995 (local address 127.21.151.62:45831)
0619 20:27:53.446831 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:53.446833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:53.446852 (+ 19us) server_negotiation.cc:407] Connection header received
0619 20:27:53.446930 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:53.446930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:53.446940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:53.447002 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:53.447343 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.447897 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.448620 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.448794 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.448842 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:53.451272 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:47995: BlockingRecv error: recv got EOF from 127.21.151.2:47995 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 20:27:53 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:53 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:53.572630 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:53.567925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:50317)
0619 20:27:53.568114 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:53.568127 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:53.568212 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:53.568395 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:53.568396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:53.568573 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:53.568620 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:53.568621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.569287 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.569288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:53.569854 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:53.569856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.570311 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.570313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:53.570387 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:53.570885 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:53.570890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:53.572555 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
W20260619 20:27:53.572705 31302 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:53.568034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:50317 (local address 127.21.151.62:45831)
0619 20:27:53.568114 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:53.568116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:53.568177 (+ 61us) server_negotiation.cc:407] Connection header received
0619 20:27:53.568276 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:53.568277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:53.568291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:53.568375 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:53.568701 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.569205 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.569931 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:53.570216 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:53.570279 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:53.572658 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:50317: BlockingRecv error: recv got EOF from 127.21.151.3:50317 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 20:27:54 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:54 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:54.296995 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:54.291769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:55709)
0619 20:27:54.291896 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:54.291911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:54.291982 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:54.292182 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:54.292183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:54.292345 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:54.292390 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:54.292392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.293022 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.293022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:54.293524 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:54.293526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.294067 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.294068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:54.294155 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:54.294760 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:54.294765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:54.296910 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
W20260619 20:27:54.297103 31306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:54.291852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:55709 (local address 127.21.151.62:45831)
0619 20:27:54.292028 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:54.292036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:54.292045 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:54.292071 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:54.292071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:54.292082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:54.292160 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:54.292457 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.292954 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.293618 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.293925 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.294023 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:54.297027 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:55709: BlockingRecv error: recv got EOF from 127.21.151.1:55709 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":125,"threads_started":1}
Jun 19 20:27:54 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:54 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:54.457008 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:54.451859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:35413)
0619 20:27:54.452085 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:54.452108 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:54.452193 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:54.452357 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:54.452358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:54.452544 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:54.452590 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:54.452592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.453277 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.453278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:54.453802 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:54.453803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.454254 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.454254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:54.454309 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:54.454874 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:54.454879 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:54.456914 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":111,"threads_started":1}
W20260619 20:27:54.457058 31306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:54.451957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:35413 (local address 127.21.151.62:45831)
0619 20:27:54.452042 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:54.452044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:54.452172 (+ 128us) server_negotiation.cc:407] Connection header received
0619 20:27:54.452261 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:54.452262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:54.452281 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:54.452350 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:54.452678 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.453181 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.453961 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.454163 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.454256 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:54.456988 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:35413: BlockingRecv error: recv got EOF from 127.21.151.2:35413 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 20:27:54 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:54 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:54.577440 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:54.573090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:36453)
0619 20:27:54.573225 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:54.573237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:54.573300 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:54.573422 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:54.573423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:54.573577 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:54.573621 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:54.573623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.574234 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.574235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:54.574734 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:54.574735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.575032 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.575033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:54.575072 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:54.575502 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:54.575506 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:54.577344 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
W20260619 20:27:54.577478 31306 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:54.573155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:36453 (local address 127.21.151.62:45831)
0619 20:27:54.573215 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:54.573216 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:54.573311 (+ 95us) server_negotiation.cc:407] Connection header received
0619 20:27:54.573349 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:54.573349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:54.573363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:54.573403 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:54.573684 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.574169 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.574800 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:54.574970 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:54.575026 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:54.577417 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:36453: BlockingRecv error: recv got EOF from 127.21.151.3:36453 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 20:27:55 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:55 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:55.302659 31309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:55.297435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:42571)
0619 20:27:55.297556 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:55.297567 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:55.297618 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:55.297805 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:55.297806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:55.298001 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:55.298064 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:55.298066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.298947 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.298948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:55.299731 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:55.299734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.300145 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.300146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:55.300205 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:55.300653 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:55.300658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:55.302551 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":50,"threads_started":1}
W20260619 20:27:55.302716 31310 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:55.297491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:42571 (local address 127.21.151.62:45831)
0619 20:27:55.297592 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:55.297594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:55.297604 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:55.297688 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:55.297689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:55.297703 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:55.297793 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:55.298163 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.298870 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.299826 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.300060 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.300132 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:55.302638 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:42571: BlockingRecv error: recv got EOF from 127.21.151.1:42571 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":51,"threads_started":1}
Jun 19 20:27:55 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:55 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:55.462005 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:55.457457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:55709)
0619 20:27:55.457606 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:55.457618 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:55.457682 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:55.457831 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:55.457832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:55.458010 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:55.458059 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:55.458060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.458756 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.458757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:55.459318 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:55.459319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.459712 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.459713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:55.459777 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:55.460207 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:55.460212 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:55.461937 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
W20260619 20:27:55.462124 31310 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:55.457534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:55709 (local address 127.21.151.62:45831)
0619 20:27:55.457628 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:55.457630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:55.457684 (+ 54us) server_negotiation.cc:407] Connection header received
0619 20:27:55.457758 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:55.457759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:55.457770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:55.457814 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:55.458127 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.458653 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.459420 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.459635 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.459713 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:55.462025 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:55709: BlockingRecv error: recv got EOF from 127.21.151.2:55709 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 20:27:55 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:55 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:55.582468 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:55.577961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:46075)
0619 20:27:55.578122 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:55.578133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:55.578197 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:55.578309 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:55.578310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:55.578464 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:55.578545 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:55.578546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.579160 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.579160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:55.579780 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:55.579782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.580117 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.580118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:55.580163 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:55.580700 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:55.580706 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:55.582398 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":111,"threads_started":1}
W20260619 20:27:55.582597 31310 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:55.577995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:46075 (local address 127.21.151.62:45831)
0619 20:27:55.578056 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:55.578058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:55.578205 (+ 147us) server_negotiation.cc:407] Connection header received
0619 20:27:55.578242 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:55.578242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:55.578255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:55.578297 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:55.578604 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.579089 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.579860 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:55.580052 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:55.580106 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:55.582496 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:46075: BlockingRecv error: recv got EOF from 127.21.151.3:46075 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 20:27:56 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:56 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:56.307984 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:56.303285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:59663)
0619 20:27:56.303430 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:56.303442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:56.303489 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:56.303692 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:56.303693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:56.303851 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:56.303899 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:56.303901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.304559 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.304561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:56.305304 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:56.305307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.305647 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.305648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:56.305695 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:56.306151 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:56.306156 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:56.307923 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
W20260619 20:27:56.308123 31314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:56.303411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:59663 (local address 127.21.151.62:45831)
0619 20:27:56.303555 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:56.303561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:56.303570 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:27:56.303591 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:56.303591 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:56.303601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:56.303673 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:56.303975 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.304468 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.305390 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.305579 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.305631 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:56.308032 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:59663: BlockingRecv error: recv got EOF from 127.21.151.1:59663 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":107,"threads_started":1}
Jun 19 20:27:56 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:56 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:56.467419 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:56.462480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:37177)
0619 20:27:56.462665 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:56.462675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:56.462731 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:56.462875 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:56.462875 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:56.463034 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:56.463080 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:56.463082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.463941 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.463943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:56.464742 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:56.464746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.465079 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.465080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:56.465117 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:56.465602 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:56.465608 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:56.467362 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
W20260619 20:27:56.467547 31314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:56.462569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:37177 (local address 127.21.151.62:45831)
0619 20:27:56.462632 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:56.462634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:56.462731 (+ 97us) server_negotiation.cc:407] Connection header received
0619 20:27:56.462800 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:56.462800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:56.462811 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:56.462855 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:56.463141 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.463828 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.464809 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.465014 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.465069 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:56.467463 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:37177: BlockingRecv error: recv got EOF from 127.21.151.2:37177 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 20:27:56 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:56 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:56.587417 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:56.583012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:40217)
0619 20:27:56.583159 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:56.583171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:56.583233 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:56.583422 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:56.583423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:56.583677 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:56.583736 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:56.583740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.584436 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.584436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:56.584967 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:56.584968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.585272 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.585273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:56.585310 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:56.585727 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:56.585732 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:56.587359 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260619 20:27:56.587517 31314 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:56.583104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:40217 (local address 127.21.151.62:45831)
0619 20:27:56.583201 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:56.583203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:56.583227 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:27:56.583313 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:56.583314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:56.583331 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:56.583391 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:56.583816 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.584367 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.585039 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:56.585212 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:56.585259 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:56.587443 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:40217: BlockingRecv error: recv got EOF from 127.21.151.3:40217 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 20:27:57 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:57 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:57.313468 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:57.308578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:41149)
0619 20:27:57.308724 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:57.308736 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:57.308802 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:57.308947 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:57.308948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:57.309112 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:57.309158 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:57.309160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.309822 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.309823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:57.310339 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:57.310341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.310785 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.310786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:57.310825 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:57.311264 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:57.311268 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:57.313361 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
W20260619 20:27:57.313508 31318 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:57.308632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:41149 (local address 127.21.151.62:45831)
0619 20:27:57.308754 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:57.308756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:57.308786 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:27:57.308856 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:57.308856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:57.308867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:57.308928 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:57.309218 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.309752 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.310479 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.310716 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.310786 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:57.313435 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:41149: BlockingRecv error: recv got EOF from 127.21.151.1:41149 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
Jun 19 20:27:57 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:57 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:57.475199 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:57.467941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:59829)
0619 20:27:57.468084 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:57.468095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:57.468164 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:57.468303 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:57.468304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:57.468465 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:57.468514 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:57.468516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.469202 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.469203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:57.469719 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:57.469720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.470057 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.470057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:57.470102 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:57.473164 (+ 3062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:57.473169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:57.475084 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
W20260619 20:27:57.475235 31318 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:57.468054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:59829 (local address 127.21.151.62:45831)
0619 20:27:57.468120 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:57.468122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:57.468146 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:27:57.468219 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:57.468220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:57.468232 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:57.468279 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:57.468576 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.469113 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.469783 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.469982 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.470042 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:57.475174 (+ 5132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:59829: BlockingRecv error: recv got EOF from 127.21.151.2:59829 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 20:27:57 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:57 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:57.592597 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:57.588114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:55695)
0619 20:27:57.588266 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:57.588283 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:57.588344 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:57.588469 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:57.588469 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:57.588643 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:57.588691 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:57.588692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.589309 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.589310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:57.589843 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:57.589844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.590142 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.590142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:57.590178 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:57.590625 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:57.590629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:57.592543 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
W20260619 20:27:57.592715 31318 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:57.588221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:55695 (local address 127.21.151.62:45831)
0619 20:27:57.588269 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:57.588271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:57.588344 (+ 73us) server_negotiation.cc:407] Connection header received
0619 20:27:57.588401 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:57.588402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:57.588415 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:57.588460 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:57.588748 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.589245 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.589907 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:57.590082 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:57.590128 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:57.592622 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:55695: BlockingRecv error: recv got EOF from 127.21.151.3:55695 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 20:27:58 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:58 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:58.318531 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:58.313926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:60835)
0619 20:27:58.314054 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:58.314064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:58.314111 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:58.314278 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:58.314279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:58.314433 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:58.314480 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:58.314481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.315160 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.315161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:58.315832 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:58.315835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.316245 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.316246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:58.316299 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:58.316742 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:58.316747 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:58.318468 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
W20260619 20:27:58.318665 31322 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:58.314007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:60835 (local address 127.21.151.62:45831)
0619 20:27:58.314104 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:58.314111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:58.314119 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:27:58.314170 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:58.314171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:58.314182 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:58.314273 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:58.314562 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.315068 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.315937 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.316171 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.316247 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:58.318555 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:60835: BlockingRecv error: recv got EOF from 127.21.151.1:60835 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
Jun 19 20:27:58 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:58 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:58.480394 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:58.475685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:43683)
0619 20:27:58.475837 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:58.475849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:58.475915 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:58.476082 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:58.476083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:58.476263 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:58.476310 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:58.476312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.476982 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.476983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:58.477539 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:58.477541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.477960 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.477961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:58.478024 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:58.478481 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:58.478485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:58.480300 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
W20260619 20:27:58.480446 31322 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:58.475759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:43683 (local address 127.21.151.62:45831)
0619 20:27:58.475822 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:58.475824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:58.475897 (+ 73us) server_negotiation.cc:407] Connection header received
0619 20:27:58.475973 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:58.475974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:58.475986 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:58.476059 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:58.476390 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.476886 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.477621 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.477814 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.477943 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:58.480384 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:43683: BlockingRecv error: recv got EOF from 127.21.151.2:43683 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 20:27:58 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:58 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:58.597770 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:58.593100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:44239)
0619 20:27:58.593275 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:58.593290 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:58.593383 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:58.593547 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:58.593548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:58.593716 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:58.593761 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:58.593763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.594451 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.594452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:58.595017 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:58.595019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.595379 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.595380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:58.595470 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:58.595957 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:58.595961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:58.597703 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
W20260619 20:27:58.597935 31322 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:58.593301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:44239 (local address 127.21.151.62:45831)
0619 20:27:58.593374 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:58.593376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:58.593386 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:27:58.593444 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:58.593445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:58.593460 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:58.593514 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:58.593849 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.594357 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.595105 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:58.595305 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:58.595382 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:58.597791 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:44239: BlockingRecv error: recv got EOF from 127.21.151.3:44239 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 20:27:59 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:59 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:59.323884 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:59.319086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:52415)
0619 20:27:59.319243 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:59.319253 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:59.319307 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:59.319453 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:59.319453 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:59.319630 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:59.319678 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:59.319680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.320308 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.320309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:59.321007 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:59.321011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.321427 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.321428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:59.321477 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:59.321980 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:59.321985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:59.323818 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
W20260619 20:27:59.324015 31326 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:59.319156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:52415 (local address 127.21.151.62:45831)
0619 20:27:59.319269 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:59.319271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:59.319295 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:27:59.319356 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:59.319356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:59.319367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:59.319425 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:59.319744 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.320234 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.321094 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.321359 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.321414 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:59.323900 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:52415: BlockingRecv error: recv got EOF from 127.21.151.1:52415 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
Jun 19 20:27:59 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:59 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:59.485725 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:59.480914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:33027)
0619 20:27:59.481116 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:59.481125 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:59.481183 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:59.481335 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:59.481335 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:59.481511 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:59.481558 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:59.481560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.482187 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.482188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:59.482998 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:59.483000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.483330 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.483330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:59.483367 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:59.483859 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:59.483863 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:59.485659 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":120,"threads_started":1}
W20260619 20:27:59.485826 31326 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:59.481018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:33027 (local address 127.21.151.62:45831)
0619 20:27:59.481084 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:59.481085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:59.481184 (+ 99us) server_negotiation.cc:407] Connection header received
0619 20:27:59.481247 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:59.481248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:59.481259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:59.481309 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:59.481620 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.482114 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.483076 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.483268 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.483321 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:59.485757 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:33027: BlockingRecv error: recv got EOF from 127.21.151.2:33027 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 20:27:59 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:27:59 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:27:59.603256 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:27:59.598405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:38137)
0619 20:27:59.598571 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 20:27:59.598583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:27:59.598641 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:27:59.598782 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:27:59.598782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:27:59.598948 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:27:59.598993 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:59.598995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.599671 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.599673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:59.600284 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:27:59.600287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.600692 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.600693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:27:59.600752 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:59.601327 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:27:59.601332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:27:59.603150 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":111,"threads_started":1}
W20260619 20:27:59.603293 31326 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:27:59.598492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:38137 (local address 127.21.151.62:45831)
0619 20:27:59.598563 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:27:59.598565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:27:59.598642 (+ 77us) server_negotiation.cc:407] Connection header received
0619 20:27:59.598706 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:27:59.598707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:27:59.598718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:27:59.598762 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:27:59.599053 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.599558 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.600362 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:27:59.600591 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:27:59.600659 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:27:59.603228 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:38137: BlockingRecv error: recv got EOF from 127.21.151.3:38137 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 20:28:00 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:00 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:00.328737 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:00.324304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:53115)
0619 20:28:00.324436 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:00.324447 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:00.324499 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:00.324622 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:00.324623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:00.324785 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:00.324830 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:00.324832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.325449 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.325450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:00.326020 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:00.326022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.326371 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.326373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:00.326420 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:00.326879 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:00.326884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:00.328682 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
W20260619 20:28:00.328881 31330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:00.324377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:53115 (local address 127.21.151.62:45831)
0619 20:28:00.324494 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:00.324496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:00.324505 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:28:00.324544 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:00.324545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:00.324555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:00.324610 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:00.324889 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.325383 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.326087 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.326283 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.326334 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:00.328762 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:53115: BlockingRecv error: recv got EOF from 127.21.151.1:53115 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":50,"threads_started":1}
Jun 19 20:28:00 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:00 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:00.490942 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:00.486286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:55121)
0619 20:28:00.486435 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:00.486448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:00.486511 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:00.486652 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:00.486653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:00.486823 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:00.486870 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:00.486872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.487560 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.487560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:00.488185 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:00.488188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.488542 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.488543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:00.488614 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:00.489080 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:00.489085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:00.490851 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
W20260619 20:28:00.491016 31330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:00.486359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:55121 (local address 127.21.151.62:45831)
0619 20:28:00.486449 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:00.486451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:00.486501 (+ 50us) server_negotiation.cc:407] Connection header received
0619 20:28:00.486558 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:00.486558 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:00.486568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:00.486621 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:00.486952 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.487478 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.488268 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.488463 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.488510 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:00.490959 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:55121: BlockingRecv error: recv got EOF from 127.21.151.2:55121 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 20:28:00 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:00 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:00.608400 31332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:00.603751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:59541)
0619 20:28:00.603872 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:00.603883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:00.603966 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:00.604140 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:00.604141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:00.604326 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:00.604369 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:00.604370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.605053 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.605054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:00.605589 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:00.605590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.605957 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.605958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:00.606032 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:00.606554 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:00.606558 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:00.608303 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
W20260619 20:28:00.608450 31330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:00.603786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:59541 (local address 127.21.151.62:45831)
0619 20:28:00.603841 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:00.603843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:00.603952 (+ 109us) server_negotiation.cc:407] Connection header received
0619 20:28:00.604045 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:00.604046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:00.604060 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:00.604135 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:00.604449 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.604947 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.605673 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:00.605870 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:00.605955 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:00.608384 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:59541: BlockingRecv error: recv got EOF from 127.21.151.3:59541 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 20:28:01 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:01 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:01.333855 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:01.329149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:47901)
0619 20:28:01.329297 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:01.329309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:01.329374 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:01.329575 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:01.329575 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:01.329744 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:01.329804 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:01.329805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.330535 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.330536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:01.331223 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:01.331226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.331540 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.331541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:01.331576 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:01.332032 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:01.332036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:01.333800 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":51,"threads_started":1}
W20260619 20:28:01.333992 31334 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:01.329344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:47901 (local address 127.21.151.62:45831)
0619 20:28:01.329462 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:01.329463 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:01.329470 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:28:01.329492 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:01.329492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:01.329502 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:01.329556 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:01.329862 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.330419 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.331284 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.331481 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.331532 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:01.333881 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:47901: BlockingRecv error: recv got EOF from 127.21.151.1:47901 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
Jun 19 20:28:01 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:01 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:01.496377 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:01.491462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:39331)
0619 20:28:01.491674 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:01.491687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:01.491757 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:01.491913 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:01.491914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:01.492110 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:01.492155 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:01.492156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.492848 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.492849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:01.493411 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:01.493436 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.493901 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.493902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:01.493982 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:01.494505 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:01.494510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:01.496282 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
W20260619 20:28:01.496428 31334 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:01.491585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:39331 (local address 127.21.151.62:45831)
0619 20:28:01.491678 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:01.491680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:01.491737 (+ 57us) server_negotiation.cc:407] Connection header received
0619 20:28:01.491817 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:01.491818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:01.491832 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:01.491907 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:01.492235 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.492732 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.493550 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.493812 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.493890 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:01.496364 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:39331: BlockingRecv error: recv got EOF from 127.21.151.2:39331 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 20:28:01 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:01 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:01.614063 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:01.609104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:57625)
0619 20:28:01.609242 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:01.609256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:01.609316 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:01.609442 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:01.609443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:01.609610 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:01.609657 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:01.609658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.610568 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.610569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:01.611143 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:01.611146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.611515 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.611516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:01.611556 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:01.612024 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:01.612029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:01.613944 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
W20260619 20:28:01.614120 31334 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:01.609155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:57625 (local address 127.21.151.62:45831)
0619 20:28:01.609235 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:01.609237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:01.609318 (+ 81us) server_negotiation.cc:407] Connection header received
0619 20:28:01.609357 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:01.609358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:01.609371 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:01.609425 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:01.609929 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.610501 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.611225 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:01.611451 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:01.611504 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:01.614047 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:57625: BlockingRecv error: recv got EOF from 127.21.151.3:57625 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 20:28:02 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:02 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:02.339093 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:02.334426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:46195)
0619 20:28:02.334601 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:02.334612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:02.334668 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:02.334828 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:02.334828 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:02.334989 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:02.335052 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:02.335053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.335791 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.335792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:02.336332 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:02.336334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.336727 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.336728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:02.336798 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:02.337312 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:02.337316 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:02.338995 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
W20260619 20:28:02.339174 31338 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:02.334516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:46195 (local address 127.21.151.62:45831)
0619 20:28:02.334651 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:02.334653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:02.334662 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:28:02.334726 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:02.334727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:02.334737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:02.334793 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:02.335140 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.335710 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.336421 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.336635 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.336724 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:02.339095 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:46195: BlockingRecv error: recv got EOF from 127.21.151.1:46195 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
Jun 19 20:28:02 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:02 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:02.505707 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:02.496952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:41119)
0619 20:28:02.497111 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:02.497162 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:02.497240 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:02.497417 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:02.497417 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:02.497614 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:02.497660 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:02.497662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.498344 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.498345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:02.498896 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:02.498898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.499285 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.499285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:02.499352 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:02.503781 (+ 4429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:02.503785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:02.505598 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
W20260619 20:28:02.505759 31338 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:02.497098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:41119 (local address 127.21.151.62:45831)
0619 20:28:02.497192 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:02.497194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:02.497222 (+ 28us) server_negotiation.cc:407] Connection header received
0619 20:28:02.497324 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:02.497326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:02.497338 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:02.497412 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:02.497760 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.498260 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.498996 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.499197 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.499283 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:02.505688 (+ 6405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:41119: BlockingRecv error: recv got EOF from 127.21.151.2:41119 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 20:28:02 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:02 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.3@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:02.619295 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:02.614556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.3:47235)
0619 20:28:02.614715 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:02.614728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:02.614801 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:02.614960 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:02.614961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:02.615141 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:02.615186 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:02.615187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.616040 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.616042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:02.616575 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:02.616577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.616976 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.616977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:02.617035 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:02.617482 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:02.617486 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:02.619182 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
W20260619 20:28:02.619345 31338 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:02.614643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:47235 (local address 127.21.151.62:45831)
0619 20:28:02.614718 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:02.614721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:02.614792 (+ 71us) server_negotiation.cc:407] Connection header received
0619 20:28:02.614866 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:02.614867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:02.614880 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:02.614926 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:02.615276 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.615952 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.616659 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:02.616868 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:02.616975 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:02.619280 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.3:47235: BlockingRecv error: recv got EOF from 127.21.151.3:47235 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 20:28:03 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:03 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.1@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:03.343916 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:03.339601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.1:54201)
0619 20:28:03.339754 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:03.339765 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:03.339814 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:03.339973 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:03.339973 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:03.340131 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:03.340177 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:03.340179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.340815 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.340815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:03.341331 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:03.341332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.341656 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.341657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:03.341697 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:03.342115 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:03.342119 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:03.343858 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":112,"threads_started":1}
W20260619 20:28:03.344081 31342 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:03.339685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:54201 (local address 127.21.151.62:45831)
0619 20:28:03.339785 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:03.339787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:03.339798 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:28:03.339877 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:03.339877 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:03.339887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:03.339967 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:03.340238 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.340748 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.341403 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.341592 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.341641 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:03.344001 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.1:54201: BlockingRecv error: recv got EOF from 127.21.151.1:54201 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":45,"threads_started":1}
Jun 19 20:28:03 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 20:28:03 dist-test-slave-f9rk krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.151.2@KRBTEST.COM for oryx/127.21.151.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 20:28:03.510807 31343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:03.506261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:45831 (local address 127.21.151.2:57649)
0619 20:28:03.506414 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:03.506448 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:03.506508 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:03.506642 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:03.506643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:03.506801 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:03.506849 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:03.506851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.507465 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.507466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:03.508070 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:03.508072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.508384 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.508385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:03.508423 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:03.508859 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:03.508864 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:03.510742 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.151.62:45831: Server oryx/127.21.151.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260619 20:28:03.510903 31342 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:03.506345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:57649 (local address 127.21.151.62:45831)
0619 20:28:03.506411 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:03.506413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:03.506510 (+ 97us) server_negotiation.cc:407] Connection header received
0619 20:28:03.506554 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:03.506554 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:03.506565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:03.506619 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:03.506911 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.507395 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.508143 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.508323 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.508371 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:03.510837 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.151.2:57649: BlockingRecv error: recv got EOF from 127.21.151.2:57649 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:28:03.544189 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 30863
I20260619 20:28:03.548774 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 31015
I20260619 20:28:03.552927 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 31150
I20260619 20:28:03.557009 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 30791
2026-06-19T20:28:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17477 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T20:28:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:28:03Z Disabled control of system clock
I20260619 20:28:03.584532 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:37893
--webserver_interface=127.21.151.62
--webserver_port=0
--builtin_ntp_servers=127.21.151.20:43291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:37893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 20:28:03.665652 31347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:03.665867 31347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:03.665889 31347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:03.667080 31347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:28:03.667122 31347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:03.667135 31347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:03.667147 31347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:28:03.667199 31347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:28:03.668603 31347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:37893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:37893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:03.668874 31347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:03.669169 31347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:03.673907 31353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:03.673975 31352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:03.674017 31347 server_base.cc:1034] running on GCE node
W20260619 20:28:03.673908 31355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:03.674449 31347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:03.674772 31347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:03.675917 31347 hybrid_clock.cc:648] HybridClock initialized: now 1781900883675902 us; error 29 us; skew 500 ppm
I20260619 20:28:03.677172 31347 webserver.cc:460] Webserver started at http://127.21.151.62:36217/ using document root <none> and password file <none>
I20260619 20:28:03.677412 31347 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:03.677485 31347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:03.677623 31347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:03.678761 31347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "81de2667471d4ba39dda31cfdbf4d536"
format_stamp: "Formatted at 2026-06-19 20:28:03 on dist-test-slave-f9rk"
server_key: "06ebc2c90f8c21f59855401819aca257"
server_key_iv: "9b61f9220b1f7e49a59da85821fd2422"
server_key_version: "encryptionkey@0"
I20260619 20:28:03.679126 31347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "81de2667471d4ba39dda31cfdbf4d536"
format_stamp: "Formatted at 2026-06-19 20:28:03 on dist-test-slave-f9rk"
server_key: "06ebc2c90f8c21f59855401819aca257"
server_key_iv: "9b61f9220b1f7e49a59da85821fd2422"
server_key_version: "encryptionkey@0"
I20260619 20:28:03.680527 31347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:28:03.681245 31361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:03.681444 31347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:28:03.681550 31347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "81de2667471d4ba39dda31cfdbf4d536"
format_stamp: "Formatted at 2026-06-19 20:28:03 on dist-test-slave-f9rk"
server_key: "06ebc2c90f8c21f59855401819aca257"
server_key_iv: "9b61f9220b1f7e49a59da85821fd2422"
server_key_version: "encryptionkey@0"
I20260619 20:28:03.681625 31347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:03.703668 31347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:03.704017 31347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:03.704182 31347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:03.708041 31347 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:37893
I20260619 20:28:03.708077 31413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:37893 every 8 connection(s)
I20260619 20:28:03.708474 31347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:28:03.708840 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 31347
I20260619 20:28:03.708943 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:28:03.709121 22108 external_mini_cluster.cc:1428] Setting key 2cc1e8e325a60bdfb27f6a323386887d
I20260619 20:28:03.709295 31414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:28:03.712853 31414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536: Bootstrap starting.
I20260619 20:28:03.713544 31414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536: Neither blocks nor log segments found. Creating new log.
I20260619 20:28:03.714203 31414 log.cc:824] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536: Log is configured to *not* fsync() on all Append() calls
I20260619 20:28:03.714828 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:03.709721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.21.151.62:37893)
0619 20:28:03.709920 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:03.709936 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:03.709962 (+ 26us) server_negotiation.cc:407] Connection header received
0619 20:28:03.710389 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:03.710395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:03.710461 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:03.710574 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:03.710808 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.711715 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.712824 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.713263 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.713352 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:03.714084 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:03.714087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:03.714088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:28:03.714251 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:03.714283 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:28:03.714313 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:03.714379 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:03.714470 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 20:28:03.714654 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260619 20:28:03.714968 31414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536: No bootstrap required, opened a new log
I20260619 20:28:03.716866 31414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81de2667471d4ba39dda31cfdbf4d536" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37893 } }
I20260619 20:28:03.717082 31414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:28:03.717183 31414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81de2667471d4ba39dda31cfdbf4d536, State: Initialized, Role: FOLLOWER
I20260619 20:28:03.717319 31414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81de2667471d4ba39dda31cfdbf4d536" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37893 } }
I20260619 20:28:03.717427 31414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:28:03.717470 31414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:28:03.717519 31414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:28:03.718061 31414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81de2667471d4ba39dda31cfdbf4d536" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37893 } }
I20260619 20:28:03.718169 31414 leader_election.cc:302] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81de2667471d4ba39dda31cfdbf4d536; no voters:
I20260619 20:28:03.718348 31414 leader_election.cc:288] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:28:03.718420 31419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:28:03.718588 31419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [term 1 LEADER]: Becoming Leader. State: Replica: 81de2667471d4ba39dda31cfdbf4d536, State: Running, Role: LEADER
I20260619 20:28:03.718632 31414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:28:03.718696 31419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81de2667471d4ba39dda31cfdbf4d536" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37893 } }
I20260619 20:28:03.719182 31421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81de2667471d4ba39dda31cfdbf4d536. Latest consensus state: current_term: 1 leader_uuid: "81de2667471d4ba39dda31cfdbf4d536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81de2667471d4ba39dda31cfdbf4d536" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37893 } } }
I20260619 20:28:03.719130 31420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81de2667471d4ba39dda31cfdbf4d536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81de2667471d4ba39dda31cfdbf4d536" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37893 } } }
I20260619 20:28:03.719300 31421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [sys.catalog]: This master's current role is: LEADER
I20260619 20:28:03.719305 31420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536 [sys.catalog]: This master's current role is: LEADER
I20260619 20:28:03.719853 31428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:28:03.720264 31428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:28:03.721940 31428 catalog_manager.cc:1347] Generated new cluster ID: 5ade3135aa0d4162a7c33fa8b448df1c
I20260619 20:28:03.722023 31428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:28:03.730479 31428 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:28:03.731088 31428 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:28:03.737278 31428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81de2667471d4ba39dda31cfdbf4d536: Generated new TSK 0
I20260619 20:28:03.737453 31428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:28:03.744958 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37893
--builtin_ntp_servers=127.21.151.20:43291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 20:28:03.827759 31438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:03.827962 31438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:03.827983 31438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:03.829205 31438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:03.829246 31438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:03.829308 31438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:28:03.830937 31438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:03.831198 31438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:03.831552 31438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:03.837608 31443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:03.837625 31444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:03.837625 31446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:03.837894 31438 server_base.cc:1034] running on GCE node
I20260619 20:28:03.838127 31438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:03.838371 31438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:03.839555 31438 hybrid_clock.cc:648] HybridClock initialized: now 1781900883839544 us; error 32 us; skew 500 ppm
I20260619 20:28:03.840965 31438 webserver.cc:460] Webserver started at http://127.21.151.1:45565/ using document root <none> and password file <none>
I20260619 20:28:03.841213 31438 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:03.841280 31438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:03.841418 31438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:03.842470 31438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "39e01424f2374d9daacc10c6d3731150"
format_stamp: "Formatted at 2026-06-19 20:28:03 on dist-test-slave-f9rk"
server_key: "67b12d001cfb1a44dce162c8986af166"
server_key_iv: "08bc564bef28406f7ae6ad6e84b19862"
server_key_version: "encryptionkey@0"
I20260619 20:28:03.842832 31438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "39e01424f2374d9daacc10c6d3731150"
format_stamp: "Formatted at 2026-06-19 20:28:03 on dist-test-slave-f9rk"
server_key: "67b12d001cfb1a44dce162c8986af166"
server_key_iv: "08bc564bef28406f7ae6ad6e84b19862"
server_key_version: "encryptionkey@0"
I20260619 20:28:03.844296 31438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:28:03.845073 31452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:03.845343 31438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:28:03.845463 31438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "39e01424f2374d9daacc10c6d3731150"
format_stamp: "Formatted at 2026-06-19 20:28:03 on dist-test-slave-f9rk"
server_key: "67b12d001cfb1a44dce162c8986af166"
server_key_iv: "08bc564bef28406f7ae6ad6e84b19862"
server_key_version: "encryptionkey@0"
I20260619 20:28:03.845528 31438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:03.865073 31438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:03.865336 31438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:03.865450 31438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:03.865638 31438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:03.865981 31438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:03.866017 31438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:03.866073 31438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:03.866099 31438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:03.871742 31438 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:42925
I20260619 20:28:03.871817 31567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:42925 every 8 connection(s)
I20260619 20:28:03.872152 31438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:28:03.877350 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:03.872909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:33149 (local address 127.21.151.62:37893)
0619 20:28:03.872974 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:03.872977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:03.873243 (+ 266us) server_negotiation.cc:407] Connection header received
0619 20:28:03.873722 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:03.873723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:03.873736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:03.873786 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:03.874419 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.874910 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.875981 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.876194 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.876257 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:03.877087 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:03.877088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:03.877089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:28:03.877093 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:03.877107 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:28:03.877111 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:03.877161 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:03.877245 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 20:28:03.877279 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:28:03.877588 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:03.872918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37893 (local address 127.21.151.1:33149)
0619 20:28:03.873123 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:03.873157 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:03.873641 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:03.873847 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:03.873850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:03.873858 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:03.874346 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:03.874350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.875004 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.875005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:03.875882 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:03.875885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:03.876273 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:03.876274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:03.876378 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:03.876952 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:28:03.876953 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:03.876981 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:28:03.877002 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:03.877171 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:03.877175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:03.877178 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:28:03.877234 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:28:03.877311 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260619 20:28:03.878126 31568 heartbeater.cc:347] Connected to a master server at 127.21.151.62:37893
I20260619 20:28:03.878225 31568 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:03.878371 31568 heartbeater.cc:511] Master 127.21.151.62:37893 requested a full tablet report, sending...
I20260619 20:28:03.878870 31378 ts_manager.cc:194] Registered new tserver with Master: 39e01424f2374d9daacc10c6d3731150 (127.21.151.1:42925)
I20260619 20:28:03.879426 31378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.1:33149
I20260619 20:28:03.880153 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 31438
I20260619 20:28:03.880270 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:28:03.880468 22108 external_mini_cluster.cc:1428] Setting key 4d9b072a36d1306ef6cb48e2b240db4c
I20260619 20:28:03.881397 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37893
--builtin_ntp_servers=127.21.151.20:43291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 20:28:03.964493 31571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:03.964694 31571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:03.964716 31571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:03.965880 31571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:03.965925 31571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:03.965952 31571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:28:03.967276 31571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:03.967533 31571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:03.967861 31571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:03.972894 31577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:03.972904 31579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:03.972904 31576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:03.973227 31571 server_base.cc:1034] running on GCE node
I20260619 20:28:03.973448 31571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:03.973693 31571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:03.974851 31571 hybrid_clock.cc:648] HybridClock initialized: now 1781900883974848 us; error 38 us; skew 500 ppm
I20260619 20:28:03.976110 31571 webserver.cc:460] Webserver started at http://127.21.151.2:39437/ using document root <none> and password file <none>
I20260619 20:28:03.976362 31571 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:03.976451 31571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:03.976593 31571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:03.977666 31571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "3007efd0155b45fea91488d61f7aeb06"
format_stamp: "Formatted at 2026-06-19 20:28:03 on dist-test-slave-f9rk"
server_key: "c38552f8aa78791dc290399a0f0e2b62"
server_key_iv: "179279d1ac9a4234721164946cb9069a"
server_key_version: "encryptionkey@0"
I20260619 20:28:03.978020 31571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "3007efd0155b45fea91488d61f7aeb06"
format_stamp: "Formatted at 2026-06-19 20:28:03 on dist-test-slave-f9rk"
server_key: "c38552f8aa78791dc290399a0f0e2b62"
server_key_iv: "179279d1ac9a4234721164946cb9069a"
server_key_version: "encryptionkey@0"
I20260619 20:28:03.979290 31571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:28:03.980062 31585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:03.980229 31571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:03.980326 31571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "3007efd0155b45fea91488d61f7aeb06"
format_stamp: "Formatted at 2026-06-19 20:28:03 on dist-test-slave-f9rk"
server_key: "c38552f8aa78791dc290399a0f0e2b62"
server_key_iv: "179279d1ac9a4234721164946cb9069a"
server_key_version: "encryptionkey@0"
I20260619 20:28:03.980404 31571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:04.003436 31571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:04.003825 31571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:04.003991 31571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:04.004225 31571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:04.004562 31571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:04.004616 31571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.004660 31571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:04.004688 31571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.010397 31571 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:36981
I20260619 20:28:04.010468 31699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:36981 every 8 connection(s)
I20260619 20:28:04.010831 31571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:28:04.016299 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.011681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:35789 (local address 127.21.151.62:37893)
0619 20:28:04.011754 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:04.011756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:04.012002 (+ 246us) server_negotiation.cc:407] Connection header received
0619 20:28:04.012549 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:04.012550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:04.012563 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:04.012624 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:04.013312 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:04.013824 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:04.014820 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:04.015033 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:04.015123 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:04.016009 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:04.016010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:04.016011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:28:04.016016 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:04.016031 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:28:04.016035 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:04.016089 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:04.016195 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 20:28:04.016219 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:28:04.016337 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 31571
I20260619 20:28:04.016450 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:28:04.016528 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.011633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37893 (local address 127.21.151.2:35789)
0619 20:28:04.011877 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:04.011912 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:04.012441 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:04.012695 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:04.012700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:04.012710 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:04.013217 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:04.013221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:04.013926 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:04.013927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:04.014726 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:04.014728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:04.015136 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:04.015137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:04.015270 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:04.015871 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:28:04.015871 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:04.015899 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:28:04.015922 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:04.016093 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:04.016098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:04.016101 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:28:04.016197 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 20:28:04.016295 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260619 20:28:04.016695 22108 external_mini_cluster.cc:1428] Setting key e9af78d280525337e8ba13b025240148
I20260619 20:28:04.017015 31700 heartbeater.cc:347] Connected to a master server at 127.21.151.62:37893
I20260619 20:28:04.017138 31700 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:04.017364 31700 heartbeater.cc:511] Master 127.21.151.62:37893 requested a full tablet report, sending...
I20260619 20:28:04.017753 31378 ts_manager.cc:194] Registered new tserver with Master: 3007efd0155b45fea91488d61f7aeb06 (127.21.151.2:36981)
I20260619 20:28:04.017896 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37893
--builtin_ntp_servers=127.21.151.20:43291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 20:28:04.018206 31378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.2:35789
W20260619 20:28:04.099917 31703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:04.100129 31703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:04.100150 31703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:04.101312 31703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:04.101353 31703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:04.101378 31703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:28:04.102782 31703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:04.103025 31703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:04.103323 31703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:04.108321 31708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:04.108314 31711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:04.108353 31709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:04.108350 31703 server_base.cc:1034] running on GCE node
I20260619 20:28:04.108781 31703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:04.109027 31703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:04.110198 31703 hybrid_clock.cc:648] HybridClock initialized: now 1781900884110187 us; error 34 us; skew 500 ppm
I20260619 20:28:04.111604 31703 webserver.cc:460] Webserver started at http://127.21.151.3:39575/ using document root <none> and password file <none>
I20260619 20:28:04.111896 31703 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:04.111945 31703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:04.112097 31703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:04.113178 31703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "1dc34083b0d1446787b277fd6059c141"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "16023b5f321abb86bc27e564a6439d74"
server_key_iv: "bf40bdcd798e434142232a9a95644e7b"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.113541 31703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "1dc34083b0d1446787b277fd6059c141"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "16023b5f321abb86bc27e564a6439d74"
server_key_iv: "bf40bdcd798e434142232a9a95644e7b"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.114861 31703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:28:04.115669 31717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.115859 31703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:28:04.115955 31703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "1dc34083b0d1446787b277fd6059c141"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "16023b5f321abb86bc27e564a6439d74"
server_key_iv: "bf40bdcd798e434142232a9a95644e7b"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.116043 31703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:04.128125 31703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:04.128435 31703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:04.128593 31703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:04.128813 31703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:04.129143 31703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:04.129197 31703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.129237 31703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:04.129264 31703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.135263 31703 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:43795
I20260619 20:28:04.135308 31830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:43795 every 8 connection(s)
I20260619 20:28:04.135701 31703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:28:04.140753 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.136411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:44435 (local address 127.21.151.62:37893)
0619 20:28:04.136500 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:04.136503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:04.136705 (+ 202us) server_negotiation.cc:407] Connection header received
0619 20:28:04.137175 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:04.137176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:04.137186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:04.137237 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:04.137882 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:04.138365 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:04.139415 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:04.139679 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:04.139734 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:04.140507 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:04.140508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:04.140509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:28:04.140513 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:04.140527 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:28:04.140531 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:04.140594 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:04.140665 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 20:28:04.140687 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 20:28:04.140988 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.136382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37893 (local address 127.21.151.3:44435)
0619 20:28:04.136587 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:04.136620 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:04.137096 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:04.137289 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:04.137292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:04.137300 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:04.137805 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:04.137810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:04.138444 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:04.138445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:04.139339 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:04.139341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:04.139755 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:04.139756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:04.139867 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:04.140350 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:28:04.140350 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:04.140378 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:28:04.140399 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:04.140596 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:04.140599 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:04.140602 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:28:04.140654 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:28:04.140732 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":110,"threads_started":1}
I20260619 20:28:04.141503 31831 heartbeater.cc:347] Connected to a master server at 127.21.151.62:37893
I20260619 20:28:04.141609 31831 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:04.141786 31831 heartbeater.cc:511] Master 127.21.151.62:37893 requested a full tablet report, sending...
I20260619 20:28:04.142156 31378 ts_manager.cc:194] Registered new tserver with Master: 1dc34083b0d1446787b277fd6059c141 (127.21.151.3:43795)
I20260619 20:28:04.142534 31378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.3:44435
I20260619 20:28:04.143136 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 31703
I20260619 20:28:04.143242 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:28:04.143453 22108 external_mini_cluster.cc:1428] Setting key 3c281175183091ac960dcf4e8c69b75e
I20260619 20:28:04.144359 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 20:28:04.146145 31417 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:04.145625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.21.151.62:37893)
0619 20:28:04.145683 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:04.145684 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:04.145807 (+ 123us) server_negotiation.cc:407] Connection header received
0619 20:28:04.145886 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:04.145886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:04.145897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:04.145951 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:04.146076 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55924: BlockingRecv error: recv got EOF from 127.0.0.1:55924 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:28:04.146184 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 31438
I20260619 20:28:04.151149 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 31571
I20260619 20:28:04.155608 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 31703
I20260619 20:28:04.159660 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 31347
2026-06-19T20:28:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (601 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T20:28:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:28:04Z Disabled control of system clock
I20260619 20:28:04.185853 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:36209
--webserver_interface=127.21.151.62
--webserver_port=0
--builtin_ntp_servers=127.21.151.20:45361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:36209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:28:04.266426 31842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:04.266639 31842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:04.266660 31842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:04.267880 31842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:28:04.267921 31842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:04.267935 31842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:04.267966 31842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:28:04.267997 31842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:28:04.269315 31842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:36209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:36209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:04.269572 31842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:04.269861 31842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:04.274591 31848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:04.274580 31847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:04.274580 31850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:04.274783 31842 server_base.cc:1034] running on GCE node
I20260619 20:28:04.275074 31842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:04.275326 31842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:04.276463 31842 hybrid_clock.cc:648] HybridClock initialized: now 1781900884276435 us; error 42 us; skew 500 ppm
I20260619 20:28:04.277647 31842 webserver.cc:460] Webserver started at http://127.21.151.62:44737/ using document root <none> and password file <none>
I20260619 20:28:04.277908 31842 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:04.277992 31842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:04.278134 31842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:04.279192 31842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "4b9d19c4e87649d89c8f0c3b612c3fec"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "d87805603f09252c82571a77405cfbef"
server_key_iv: "ebe8b3fa3993ff8ed5cbe313fb315a6f"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.279553 31842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "4b9d19c4e87649d89c8f0c3b612c3fec"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "d87805603f09252c82571a77405cfbef"
server_key_iv: "ebe8b3fa3993ff8ed5cbe313fb315a6f"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.280892 31842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:28:04.281592 31856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.281776 31842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:28:04.281862 31842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "4b9d19c4e87649d89c8f0c3b612c3fec"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "d87805603f09252c82571a77405cfbef"
server_key_iv: "ebe8b3fa3993ff8ed5cbe313fb315a6f"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.281951 31842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:04.285302 31842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:04.285544 31842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:04.285701 31842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:04.289462 31842 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:36209
I20260619 20:28:04.289489 31908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:36209 every 8 connection(s)
I20260619 20:28:04.289803 31842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:28:04.290457 31909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:28:04.293196 31909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec: Bootstrap starting.
I20260619 20:28:04.293753 31909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec: Neither blocks nor log segments found. Creating new log.
I20260619 20:28:04.294057 31909 log.cc:824] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec: Log is configured to *not* fsync() on all Append() calls
I20260619 20:28:04.294693 31909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec: No bootstrap required, opened a new log
I20260619 20:28:04.296320 31909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9d19c4e87649d89c8f0c3b612c3fec" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36209 } }
I20260619 20:28:04.296452 31909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:28:04.296474 31909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b9d19c4e87649d89c8f0c3b612c3fec, State: Initialized, Role: FOLLOWER
I20260619 20:28:04.296663 31909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9d19c4e87649d89c8f0c3b612c3fec" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36209 } }
I20260619 20:28:04.296736 31909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:28:04.296774 31909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:28:04.296823 31909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:28:04.297328 31909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9d19c4e87649d89c8f0c3b612c3fec" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36209 } }
I20260619 20:28:04.297433 31909 leader_election.cc:302] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b9d19c4e87649d89c8f0c3b612c3fec; no voters:
I20260619 20:28:04.297610 31909 leader_election.cc:288] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:28:04.297648 31912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:28:04.297802 31912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [term 1 LEADER]: Becoming Leader. State: Replica: 4b9d19c4e87649d89c8f0c3b612c3fec, State: Running, Role: LEADER
I20260619 20:28:04.297849 31909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:28:04.297887 31912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9d19c4e87649d89c8f0c3b612c3fec" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36209 } }
I20260619 20:28:04.298229 31913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b9d19c4e87649d89c8f0c3b612c3fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9d19c4e87649d89c8f0c3b612c3fec" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36209 } } }
I20260619 20:28:04.298391 31913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [sys.catalog]: This master's current role is: LEADER
I20260619 20:28:04.298480 31914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b9d19c4e87649d89c8f0c3b612c3fec. Latest consensus state: current_term: 1 leader_uuid: "4b9d19c4e87649d89c8f0c3b612c3fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b9d19c4e87649d89c8f0c3b612c3fec" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36209 } } }
I20260619 20:28:04.298557 31914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec [sys.catalog]: This master's current role is: LEADER
I20260619 20:28:04.298830 31921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:28:04.299331 31921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:28:04.299860 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 31842
I20260619 20:28:04.299962 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:28:04.300148 22108 external_mini_cluster.cc:1428] Setting key f2522f4a15230f06a87d305d6a76d1c5
I20260619 20:28:04.300889 31921 catalog_manager.cc:1347] Generated new cluster ID: 833e296ed62e415289f5600bc8002309
I20260619 20:28:04.300951 31921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:28:04.302498 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.300613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.21.151.62:36209)
0619 20:28:04.301035 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:04.301043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:04.301079 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:28:04.301166 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:04.301171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:04.301204 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:04.301330 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:04.302035 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:04.302038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:04.302038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:28:04.302042 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:04.302086 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:28:04.302102 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:04.302172 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:04.302251 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 20:28:04.302313 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":66,"threads_started":1}
I20260619 20:28:04.316023 31921 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:28:04.316185 31921 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:28:04.333549 31921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b9d19c4e87649d89c8f0c3b612c3fec: Generated new TSK 0
I20260619 20:28:04.333760 31921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:28:04.337066 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:36209
--builtin_ntp_servers=127.21.151.20:45361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:28:04.427812 31933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:04.428038 31933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:04.428072 31933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:04.429270 31933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:04.429356 31933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:04.429399 31933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:28:04.430732 31933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:36209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:04.430997 31933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:04.431282 31933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:04.437074 31938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:04.437074 31941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:04.437088 31939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:04.437211 31933 server_base.cc:1034] running on GCE node
I20260619 20:28:04.437695 31933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:04.437933 31933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:04.439141 31933 hybrid_clock.cc:648] HybridClock initialized: now 1781900884439127 us; error 30 us; skew 500 ppm
I20260619 20:28:04.440513 31933 webserver.cc:460] Webserver started at http://127.21.151.1:38097/ using document root <none> and password file <none>
I20260619 20:28:04.440784 31933 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:04.440853 31933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:04.441009 31933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:04.442178 31933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "e5e1a9c46aad4e6fbd185b7ca7d94f80"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "1346e6e9bc00d4d93746fffe194eef24"
server_key_iv: "3170b915a22a9eebb8e40175bad286c0"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.442517 31933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "e5e1a9c46aad4e6fbd185b7ca7d94f80"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "1346e6e9bc00d4d93746fffe194eef24"
server_key_iv: "3170b915a22a9eebb8e40175bad286c0"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.443820 31933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:28:04.444567 31947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.444775 31933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:28:04.444876 31933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "e5e1a9c46aad4e6fbd185b7ca7d94f80"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "1346e6e9bc00d4d93746fffe194eef24"
server_key_iv: "3170b915a22a9eebb8e40175bad286c0"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.444949 31933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:04.449605 31933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:04.449877 31933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:04.450007 31933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:04.450198 31933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:04.450505 31933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:04.450549 31933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.450575 31933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:04.450591 31933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.456276 31933 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:37635
I20260619 20:28:04.456353 32060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:37635 every 8 connection(s)
I20260619 20:28:04.456662 31933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:28:04.459530 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.457409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:41135 (local address 127.21.151.62:36209)
0619 20:28:04.457448 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:04.457450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:04.457676 (+ 226us) server_negotiation.cc:407] Connection header received
0619 20:28:04.458161 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:04.458162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:04.458170 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:04.458225 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:04.459298 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:04.459299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:04.459299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:28:04.459304 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:04.459319 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:28:04.459323 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:04.459370 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:04.459442 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 20:28:04.459457 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:28:04.459730 32063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.457389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:36209 (local address 127.21.151.1:41135)
0619 20:28:04.457564 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:04.457595 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:04.458059 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:04.458291 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:04.458294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:04.458303 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:04.459148 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:28:04.459149 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:04.459175 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:28:04.459205 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:04.459368 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:04.459372 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:04.459375 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:28:04.459450 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:28:04.459505 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260619 20:28:04.460181 32061 heartbeater.cc:347] Connected to a master server at 127.21.151.62:36209
I20260619 20:28:04.460265 32061 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:04.460427 32061 heartbeater.cc:511] Master 127.21.151.62:36209 requested a full tablet report, sending...
I20260619 20:28:04.461047 31873 ts_manager.cc:194] Registered new tserver with Master: e5e1a9c46aad4e6fbd185b7ca7d94f80 (127.21.151.1:37635)
I20260619 20:28:04.462451 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 31933
I20260619 20:28:04.462551 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:28:04.462743 22108 external_mini_cluster.cc:1428] Setting key 396cccc3962afef31d6cd5d43364c50e
I20260619 20:28:04.463721 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:36209
--builtin_ntp_servers=127.21.151.20:45361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:28:04.545760 32064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:04.545945 32064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:04.545966 32064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:04.547138 32064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:04.547176 32064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:04.547230 32064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:28:04.549093 32064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:36209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:04.549340 32064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:04.549624 32064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:04.554451 32069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:04.554450 32072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:04.554522 32070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:04.554607 32064 server_base.cc:1034] running on GCE node
I20260619 20:28:04.554806 32064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:04.555050 32064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:04.556185 32064 hybrid_clock.cc:648] HybridClock initialized: now 1781900884556173 us; error 29 us; skew 500 ppm
I20260619 20:28:04.557351 32064 webserver.cc:460] Webserver started at http://127.21.151.2:32909/ using document root <none> and password file <none>
I20260619 20:28:04.557595 32064 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:04.557641 32064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:04.557812 32064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:04.558844 32064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "d5687e117a284fe9b1c8f290d7918641"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "59eee5900f143e5fb2c142e1bc96490f"
server_key_iv: "bbd6f0ff09b5325abcc60f173a591007"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.559198 32064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "d5687e117a284fe9b1c8f290d7918641"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "59eee5900f143e5fb2c142e1bc96490f"
server_key_iv: "bbd6f0ff09b5325abcc60f173a591007"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.560539 32064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:28:04.561254 32078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.561421 32064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:28:04.561515 32064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "d5687e117a284fe9b1c8f290d7918641"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "59eee5900f143e5fb2c142e1bc96490f"
server_key_iv: "bbd6f0ff09b5325abcc60f173a591007"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.561597 32064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:04.564901 32064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:04.565157 32064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:04.565303 32064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:04.565507 32064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:04.565793 32064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:04.565845 32064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.565884 32064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:04.565912 32064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.571493 32064 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:34123
I20260619 20:28:04.571544 32191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:34123 every 8 connection(s)
I20260619 20:28:04.571856 32064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:28:04.574800 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.572507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:43295 (local address 127.21.151.62:36209)
0619 20:28:04.572568 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:04.572570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:04.572810 (+ 240us) server_negotiation.cc:407] Connection header received
0619 20:28:04.573338 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:04.573339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:04.573345 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:04.573402 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:04.574502 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:04.574503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:04.574503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:28:04.574508 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:04.574526 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:28:04.574530 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:04.574616 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:04.574713 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 20:28:04.574729 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:28:04.574991 32194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.572497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:36209 (local address 127.21.151.2:43295)
0619 20:28:04.572689 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:04.572721 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:04.573227 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:04.573478 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:04.573481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:04.573490 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:04.574343 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:28:04.574343 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:04.574369 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:28:04.574396 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:04.574611 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:04.574616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:04.574620 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:28:04.574710 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 20:28:04.574770 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260619 20:28:04.575500 32192 heartbeater.cc:347] Connected to a master server at 127.21.151.62:36209
I20260619 20:28:04.575609 32192 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:04.575793 32192 heartbeater.cc:511] Master 127.21.151.62:36209 requested a full tablet report, sending...
I20260619 20:28:04.576160 31873 ts_manager.cc:194] Registered new tserver with Master: d5687e117a284fe9b1c8f290d7918641 (127.21.151.2:34123)
I20260619 20:28:04.578154 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 32064
I20260619 20:28:04.578253 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:28:04.578434 22108 external_mini_cluster.cc:1428] Setting key 73c4cfba253e147598eb68cb96bc6325
I20260619 20:28:04.579272 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:36209
--builtin_ntp_servers=127.21.151.20:45361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 20:28:04.658638 32195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:04.658856 32195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:04.658890 32195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:04.660094 32195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:04.660156 32195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:04.660197 32195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:28:04.661540 32195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:36209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:04.661799 32195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:04.662089 32195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:04.667009 32200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:04.667027 32201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:04.667006 32203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:04.667068 32195 server_base.cc:1034] running on GCE node
I20260619 20:28:04.667397 32195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:04.667706 32195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:04.668846 32195 hybrid_clock.cc:648] HybridClock initialized: now 1781900884668830 us; error 33 us; skew 500 ppm
I20260619 20:28:04.670006 32195 webserver.cc:460] Webserver started at http://127.21.151.3:40019/ using document root <none> and password file <none>
I20260619 20:28:04.670239 32195 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:04.670285 32195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:04.670445 32195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:04.671712 32195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "0b83efce6a9541b6a4d6738e8bda2828"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "cb66f5c71870316b3fa100b9a25fc6cb"
server_key_iv: "ee11bcbe5cbbff8d222853c0b1ddbacf"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.672070 32195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "0b83efce6a9541b6a4d6738e8bda2828"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "cb66f5c71870316b3fa100b9a25fc6cb"
server_key_iv: "ee11bcbe5cbbff8d222853c0b1ddbacf"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.673326 32195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:28:04.674047 32209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.674222 32195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:28:04.674309 32195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "0b83efce6a9541b6a4d6738e8bda2828"
format_stamp: "Formatted at 2026-06-19 20:28:04 on dist-test-slave-f9rk"
server_key: "cb66f5c71870316b3fa100b9a25fc6cb"
server_key_iv: "ee11bcbe5cbbff8d222853c0b1ddbacf"
server_key_version: "encryptionkey@0"
I20260619 20:28:04.674379 32195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:04.678282 32195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:04.678532 32195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:04.678659 32195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:04.678851 32195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:04.679145 32195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:04.679176 32195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.679229 32195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:04.679255 32195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:04.684818 32195 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:36235
I20260619 20:28:04.684890 32323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:36235 every 8 connection(s)
I20260619 20:28:04.685225 32195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:28:04.688139 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.685980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:52755 (local address 127.21.151.62:36209)
0619 20:28:04.686024 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:04.686026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:04.686256 (+ 230us) server_negotiation.cc:407] Connection header received
0619 20:28:04.686702 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:04.686703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:04.686708 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:04.686758 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:04.687940 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:04.687941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:04.687941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:28:04.687945 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:04.687959 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:28:04.687963 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:04.688009 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:04.688076 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 20:28:04.688088 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:28:04.688361 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:04.685975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:36209 (local address 127.21.151.3:52755)
0619 20:28:04.686143 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:04.686175 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:04.686618 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:04.686809 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:04.686812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:04.686820 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:04.687807 (+ 987us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:28:04.687807 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:04.687834 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:28:04.687861 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:04.688008 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:04.688010 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:04.688013 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:28:04.688076 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 20:28:04.688122 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260619 20:28:04.688815 32324 heartbeater.cc:347] Connected to a master server at 127.21.151.62:36209
I20260619 20:28:04.688920 32324 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:04.689033 32324 heartbeater.cc:511] Master 127.21.151.62:36209 requested a full tablet report, sending...
I20260619 20:28:04.689345 31873 ts_manager.cc:194] Registered new tserver with Master: 0b83efce6a9541b6a4d6738e8bda2828 (127.21.151.3:36235)
I20260619 20:28:04.693883 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 32195
I20260619 20:28:04.693984 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:28:04.694176 22108 external_mini_cluster.cc:1428] Setting key e14cdfed325a1b41158b2a938875ece1
I20260619 20:28:04.695087 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 20:28:04.696982 31932 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:28:04.696517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.21.151.62:36209)
0619 20:28:04.696556 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:04.696557 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:04.696646 (+ 89us) server_negotiation.cc:407] Connection header received
0619 20:28:04.696710 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:04.696710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:04.696714 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:04.696761 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:04.696896 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47392: BlockingRecv error: recv got EOF from 127.0.0.1:47392 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:28:04.697115 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 31933
I20260619 20:28:04.702777 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 32064
I20260619 20:28:04.707135 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 32195
I20260619 20:28:04.711450 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 31842
2026-06-19T20:28:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (551 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:28:04 dist-test-slave-f9rk krb5kdc[32334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:28:04 dist-test-slave-f9rk krb5kdc[32334](info): set up 2 sockets
Jun 19 20:28:04 dist-test-slave-f9rk krb5kdc[32334](info): commencing operation
krb5kdc: starting...
W20260619 20:28:05.757771 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:28:05 dist-test-slave-f9rk krb5kdc[32334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:28:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:28:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:28:05.872121 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:39983
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:39983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:28:05.951414 32348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:05.951608 32348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:05.951653 32348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:05.952761 32348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:28:05.952800 32348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:05.952814 32348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:05.952826 32348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:28:05.952878 32348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:28:05.954334 32348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:39983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:39983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:05.954571 32348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:05.954787 32348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:05.959265 32354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:05.959272 32356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:05.959354 32348 server_base.cc:1034] running on GCE node
W20260619 20:28:05.959690 32353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:05.960011 32348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:05.960289 32348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:05.961428 32348 hybrid_clock.cc:648] HybridClock initialized: now 1781900885961402 us; error 48 us; skew 500 ppm
Jun 19 20:28:05 dist-test-slave-f9rk krb5kdc[32334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900885, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:28:05.964896 32348 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:28:05.965405 32348 webserver.cc:460] Webserver started at http://127.21.151.62:34129/ using document root <none> and password file <none>
I20260619 20:28:05.965632 32348 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:05.965698 32348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:05.965890 32348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:05.967191 32348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "fd74c52800504b1db735c6544f417432"
format_stamp: "Formatted at 2026-06-19 20:28:05 on dist-test-slave-f9rk"
server_key: "01b3a19b3d6274710c3a536a5a15bb52"
server_key_iv: "9d3d96e4edc0e3729d755771dab49ba1"
server_key_version: "encryptionkey@0"
I20260619 20:28:05.967574 32348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "fd74c52800504b1db735c6544f417432"
format_stamp: "Formatted at 2026-06-19 20:28:05 on dist-test-slave-f9rk"
server_key: "01b3a19b3d6274710c3a536a5a15bb52"
server_key_iv: "9d3d96e4edc0e3729d755771dab49ba1"
server_key_version: "encryptionkey@0"
I20260619 20:28:05.969182 32348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:28:05.970115 32363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:05.970283 32348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:28:05.970418 32348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "fd74c52800504b1db735c6544f417432"
format_stamp: "Formatted at 2026-06-19 20:28:05 on dist-test-slave-f9rk"
server_key: "01b3a19b3d6274710c3a536a5a15bb52"
server_key_iv: "9d3d96e4edc0e3729d755771dab49ba1"
server_key_version: "encryptionkey@0"
I20260619 20:28:05.970515 32348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:05.988327 32348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:05.990579 32348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:05.990741 32348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:05.994625 32348 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:39983
I20260619 20:28:05.994660 32415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:39983 every 8 connection(s)
I20260619 20:28:05.995035 32348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:28:05.996078 32416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:28:05.996381 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 32348
I20260619 20:28:05.996495 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:28:05.996711 22108 external_mini_cluster.cc:1428] Setting key 2b998bb117485e5b26107940703f9178
I20260619 20:28:05.998594 32416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432: Bootstrap starting.
I20260619 20:28:05.999166 32416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432: Neither blocks nor log segments found. Creating new log.
I20260619 20:28:05.999452 32416 log.cc:824] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432: Log is configured to *not* fsync() on all Append() calls
I20260619 20:28:06.000262 32416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432: No bootstrap required, opened a new log
I20260619 20:28:06.002635 32416 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd74c52800504b1db735c6544f417432" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39983 } }
I20260619 20:28:06.002820 32416 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:28:06.002852 32416 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd74c52800504b1db735c6544f417432, State: Initialized, Role: FOLLOWER
I20260619 20:28:06.003001 32416 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd74c52800504b1db735c6544f417432" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39983 } }
I20260619 20:28:06.003110 32416 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:28:06.003157 32416 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:28:06.003232 32416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 20:28:05 dist-test-slave-f9rk krb5kdc[32334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:28:06.003908 32416 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd74c52800504b1db735c6544f417432" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39983 } }
I20260619 20:28:06.004036 32416 leader_election.cc:302] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd74c52800504b1db735c6544f417432; no voters:
I20260619 20:28:06.004204 32416 leader_election.cc:288] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:28:06.004248 32421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:28:06.004463 32421 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [term 1 LEADER]: Becoming Leader. State: Replica: fd74c52800504b1db735c6544f417432, State: Running, Role: LEADER
I20260619 20:28:06.004498 32416 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:28:06.004626 32421 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd74c52800504b1db735c6544f417432" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39983 } }
I20260619 20:28:06.004999 32422 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd74c52800504b1db735c6544f417432" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd74c52800504b1db735c6544f417432" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39983 } } }
I20260619 20:28:06.005005 32423 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd74c52800504b1db735c6544f417432. Latest consensus state: current_term: 1 leader_uuid: "fd74c52800504b1db735c6544f417432" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd74c52800504b1db735c6544f417432" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39983 } } }
I20260619 20:28:06.005087 32422 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [sys.catalog]: This master's current role is: LEADER
I20260619 20:28:06.005111 32423 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432 [sys.catalog]: This master's current role is: LEADER
I20260619 20:28:06.005875 32427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:28:06.006371 32427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:28:06.007975 32427 catalog_manager.cc:1347] Generated new cluster ID: c8333fe332e247ba9e417edaa699274d
I20260619 20:28:06.008025 32427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:28:06.015061 32427 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:28:06.015785 32427 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:28:06.016563 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:05.997259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.21.151.62:39983)
0619 20:28:05.997449 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:05.997462 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:05.997488 (+ 26us) server_negotiation.cc:407] Connection header received
0619 20:28:05.997875 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:05.997880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:05.997927 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:05.998102 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:05.998744 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:05.999841 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.000919 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.001366 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.001452 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.004252 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:06.004257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:06.004257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:28:06.004270 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:06.014946 (+ 10676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:06.015363 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:06.015366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:06.015367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:06.015459 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:06.015706 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:06.015707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:06.015707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:06.015980 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:06.016163 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.016231 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.016385 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260619 20:28:06.020677 32427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432: Generated new TSK 0
I20260619 20:28:06.020865 32427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:28:06.050174 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39983
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:28:06.146739 32442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:06.146946 32442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:06.146970 32442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:06.148182 32442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:06.148222 32442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:06.148249 32442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:28:06.149693 32442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:06.149956 32442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:06.150229 32442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:06.155102 32450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:06.155105 32447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:06.155342 32442 server_base.cc:1034] running on GCE node
W20260619 20:28:06.155407 32448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:06.155691 32442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:06.156009 32442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:06.157166 32442 hybrid_clock.cc:648] HybridClock initialized: now 1781900886157155 us; error 35 us; skew 500 ppm
Jun 19 20:28:06 dist-test-slave-f9rk krb5kdc[32334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900886, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:28:06.159744 32442 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:28:06.160166 32442 webserver.cc:460] Webserver started at http://127.21.151.1:44213/ using document root <none> and password file <none>
I20260619 20:28:06.160410 32442 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:06.160487 32442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:06.160647 32442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:06.161684 32442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "2cdffb7805344fb0b6702947bc60bb82"
format_stamp: "Formatted at 2026-06-19 20:28:06 on dist-test-slave-f9rk"
server_key: "fb99e56b4fb616d3f728069b80335a03"
server_key_iv: "6e227c369dc62ec2789a1bea295cf296"
server_key_version: "encryptionkey@0"
I20260619 20:28:06.162046 32442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "2cdffb7805344fb0b6702947bc60bb82"
format_stamp: "Formatted at 2026-06-19 20:28:06 on dist-test-slave-f9rk"
server_key: "fb99e56b4fb616d3f728069b80335a03"
server_key_iv: "6e227c369dc62ec2789a1bea295cf296"
server_key_version: "encryptionkey@0"
I20260619 20:28:06.163374 32442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:28:06.164233 32457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.164400 32442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.164508 32442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "2cdffb7805344fb0b6702947bc60bb82"
format_stamp: "Formatted at 2026-06-19 20:28:06 on dist-test-slave-f9rk"
server_key: "fb99e56b4fb616d3f728069b80335a03"
server_key_iv: "6e227c369dc62ec2789a1bea295cf296"
server_key_version: "encryptionkey@0"
I20260619 20:28:06.164592 32442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:06.185243 32442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:06.187712 32442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:06.187878 32442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:06.188136 32442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:06.188457 32442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:06.188510 32442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.188555 32442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:06.188577 32442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.194151 32442 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:44817
I20260619 20:28:06.194231 32571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:44817 every 8 connection(s)
I20260619 20:28:06.194551 32442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:28:06.195729 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 32442
I20260619 20:28:06.195807 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:28:06.196028 22108 external_mini_cluster.cc:1428] Setting key d1b3cf41659c3cf9dd022cb1aa197029
Jun 19 20:28:06 dist-test-slave-f9rk krb5kdc[32334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900886, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
I20260619 20:28:06.204457 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.195278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:36443 (local address 127.21.151.62:39983)
0619 20:28:06.195358 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:06.195360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:06.195588 (+ 228us) server_negotiation.cc:407] Connection header received
0619 20:28:06.196263 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:06.196265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:06.196278 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:06.196323 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:06.197272 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.197791 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.198810 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.199041 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.199131 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.201559 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:06.201561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:06.201561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:28:06.201570 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:06.202968 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:06.203385 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:06.203386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:06.203386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:06.203456 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:06.203787 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:06.203788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:06.203789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:06.203928 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:06.204000 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.204242 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.204319 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 20:28:06.204631 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.195252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39983 (local address 127.21.151.1:36443)
0619 20:28:06.195460 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:06.195497 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:06.196144 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:06.196392 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:06.196396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:06.196973 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:06.197168 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.197172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.197893 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.197894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.198713 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.198715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.199142 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.199143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.199272 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.199873 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:06.199884 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:06.201464 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:06.203070 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:06.203077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:06.203079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:06.203306 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:06.203546 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:06.203546 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:06.203547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:06.203714 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:06.203996 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:06.204001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:06.204117 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 20:28:06.204195 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 20:28:06.204316 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":122,"threads_started":1}
I20260619 20:28:06.205135 32572 heartbeater.cc:347] Connected to a master server at 127.21.151.62:39983
I20260619 20:28:06.205368 32572 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:06.205566 32572 heartbeater.cc:511] Master 127.21.151.62:39983 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
I20260619 20:28:06.206188 32380 ts_manager.cc:194] Registered new tserver with Master: 2cdffb7805344fb0b6702947bc60bb82 (127.21.151.1:44817)
I20260619 20:28:06.207165 32380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:36443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:28:06.219592 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39983
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:28:06.300606 32577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:06.300813 32577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:06.300836 32577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:06.301921 32577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:06.301954 32577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:06.301980 32577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:28:06.303328 32577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:06.303524 32577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:06.303784 32577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:06.308704 32582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:06.308702 32583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:06.308697 32585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:06.308887 32577 server_base.cc:1034] running on GCE node
I20260619 20:28:06.309271 32577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:06.309511 32577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:06.310683 32577 hybrid_clock.cc:648] HybridClock initialized: now 1781900886310685 us; error 42 us; skew 500 ppm
Jun 19 20:28:06 dist-test-slave-f9rk krb5kdc[32334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900886, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:28:06.313138 32577 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:28:06.313603 32577 webserver.cc:460] Webserver started at http://127.21.151.2:37897/ using document root <none> and password file <none>
I20260619 20:28:06.313881 32577 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:06.313957 32577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:06.314092 32577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:06.315106 32577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "2b61e329a4144494b26c5b29a06d023e"
format_stamp: "Formatted at 2026-06-19 20:28:06 on dist-test-slave-f9rk"
server_key: "0725c45a47256d360a76b96eb81fd41f"
server_key_iv: "2d8f7483865e3b0ecdca8ac85ac42ce4"
server_key_version: "encryptionkey@0"
I20260619 20:28:06.315460 32577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "2b61e329a4144494b26c5b29a06d023e"
format_stamp: "Formatted at 2026-06-19 20:28:06 on dist-test-slave-f9rk"
server_key: "0725c45a47256d360a76b96eb81fd41f"
server_key_iv: "2d8f7483865e3b0ecdca8ac85ac42ce4"
server_key_version: "encryptionkey@0"
I20260619 20:28:06.316741 32577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:28:06.317500 32592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.317672 32577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:28:06.317765 32577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "2b61e329a4144494b26c5b29a06d023e"
format_stamp: "Formatted at 2026-06-19 20:28:06 on dist-test-slave-f9rk"
server_key: "0725c45a47256d360a76b96eb81fd41f"
server_key_iv: "2d8f7483865e3b0ecdca8ac85ac42ce4"
server_key_version: "encryptionkey@0"
I20260619 20:28:06.317857 32577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:06.333371 32577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:06.335175 32577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:06.335330 32577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:06.335549 32577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:06.335855 32577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:06.335907 32577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.335942 32577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:06.335970 32577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.341698 32577 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:32917
I20260619 20:28:06.341737 32706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:32917 every 8 connection(s)
I20260619 20:28:06.342080 32577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:28:06.344372 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 32577
I20260619 20:28:06.344480 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:28:06.344674 22108 external_mini_cluster.cc:1428] Setting key 2d0fee706d0f471c205c93449235fe35
Jun 19 20:28:06 dist-test-slave-f9rk krb5kdc[32334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900886, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:28:06.352746 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.342795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:34657 (local address 127.21.151.62:39983)
0619 20:28:06.342852 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:06.342854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:06.343030 (+ 176us) server_negotiation.cc:407] Connection header received
0619 20:28:06.343536 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:06.343537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:06.343548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:06.343661 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:06.344852 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.345473 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.346922 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.347267 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.347343 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.349684 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:06.349685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:06.349685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:28:06.349691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:06.351071 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:06.351809 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:06.351810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:06.351810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:06.351861 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:06.352162 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:06.352162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:06.352163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:06.352256 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:06.352301 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.352625 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.352684 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:28:06.353029 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.342777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39983 (local address 127.21.151.2:34657)
0619 20:28:06.342913 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:06.342950 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:06.343435 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:06.343750 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:06.343753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:06.344405 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:06.344705 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.344712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.345558 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.345559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.346833 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.346836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.347351 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.347351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.347446 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.347966 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:06.347974 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:06.349598 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:06.351154 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:06.351174 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:06.351177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:06.351487 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:06.351940 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:06.351941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:06.351942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:06.352080 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:06.352320 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:06.352326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:06.352482 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 20:28:06.352595 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 20:28:06.352762 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
I20260619 20:28:06.353579 32707 heartbeater.cc:347] Connected to a master server at 127.21.151.62:39983
I20260619 20:28:06.353688 32707 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:06.353876 32707 heartbeater.cc:511] Master 127.21.151.62:39983 requested a full tablet report, sending...
I20260619 20:28:06.354282 32380 ts_manager.cc:194] Registered new tserver with Master: 2b61e329a4144494b26c5b29a06d023e (127.21.151.2:32917)
I20260619 20:28:06.354676 32380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:34657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:28:06.368382 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39983
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:28:06.447762 32712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:06.447989 32712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:06.448027 32712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:06.449196 32712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:06.449270 32712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:06.449313 32712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:28:06.450738 32712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:06.451035 32712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:06.451259 32712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:06.456337 32717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:06.456399 32718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:06.456498 32712 server_base.cc:1034] running on GCE node
W20260619 20:28:06.456395 32720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:06.456750 32712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:06.456986 32712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:06.458130 32712 hybrid_clock.cc:648] HybridClock initialized: now 1781900886458128 us; error 37 us; skew 500 ppm
Jun 19 20:28:06 dist-test-slave-f9rk krb5kdc[32334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900886, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:28:06.460556 32712 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:28:06.460970 32712 webserver.cc:460] Webserver started at http://127.21.151.3:38111/ using document root <none> and password file <none>
I20260619 20:28:06.461187 32712 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:06.461251 32712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:06.461397 32712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:28:06.462394 32712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "151e657b669044f1b6590fb9807b2e26"
format_stamp: "Formatted at 2026-06-19 20:28:06 on dist-test-slave-f9rk"
server_key: "fafd2e7383bb603eec0be80e638c239a"
server_key_iv: "b28225d19cb9f200c802ba1f8a37d892"
server_key_version: "encryptionkey@0"
I20260619 20:28:06.462764 32712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "151e657b669044f1b6590fb9807b2e26"
format_stamp: "Formatted at 2026-06-19 20:28:06 on dist-test-slave-f9rk"
server_key: "fafd2e7383bb603eec0be80e638c239a"
server_key_iv: "b28225d19cb9f200c802ba1f8a37d892"
server_key_version: "encryptionkey@0"
I20260619 20:28:06.464092 32712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:28:06.464792 32727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.464952 32712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.465051 32712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "151e657b669044f1b6590fb9807b2e26"
format_stamp: "Formatted at 2026-06-19 20:28:06 on dist-test-slave-f9rk"
server_key: "fafd2e7383bb603eec0be80e638c239a"
server_key_iv: "b28225d19cb9f200c802ba1f8a37d892"
server_key_version: "encryptionkey@0"
I20260619 20:28:06.465122 32712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:06.488898 32712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:06.490811 32712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:06.490976 32712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:06.491216 32712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:06.491536 32712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:06.491591 32712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.491647 32712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:06.491676 32712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:06.497416 32712 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:43545
I20260619 20:28:06.497473 375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:43545 every 8 connection(s)
I20260619 20:28:06.497777 32712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:28:06.503126 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 32712
I20260619 20:28:06.503250 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:28:06.503432 22108 external_mini_cluster.cc:1428] Setting key d0d70459a9914a14c621c22449a609b0
Jun 19 20:28:06 dist-test-slave-f9rk krb5kdc[32334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900886, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:28:06.507321 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.498477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:49441 (local address 127.21.151.62:39983)
0619 20:28:06.498566 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:06.498569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:06.498780 (+ 211us) server_negotiation.cc:407] Connection header received
0619 20:28:06.499304 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:06.499305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:06.499318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:06.499378 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:06.500391 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.500908 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.501904 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.502102 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.502203 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.504431 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:06.504432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:06.504433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:28:06.504440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:06.505866 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:06.506375 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:06.506376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:06.506376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:06.506435 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:06.506686 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:06.506687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:06.506687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:06.506798 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:06.506876 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.507149 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.507226 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 20:28:06.507542 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.498424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39983 (local address 127.21.151.3:49441)
0619 20:28:06.498674 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:06.498703 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:06.499199 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:06.499451 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:06.499455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:06.500101 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:06.500285 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.500290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.501010 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.501011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.501799 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.501801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.502211 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.502212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.502333 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.502742 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:06.502750 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:06.504327 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:06.505982 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:06.505988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:06.505991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:06.506298 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:06.506505 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:06.506506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:06.506507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:06.506619 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:06.506866 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:06.506872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:06.507011 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 20:28:06.507128 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 20:28:06.507289 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
I20260619 20:28:06.507998 376 heartbeater.cc:347] Connected to a master server at 127.21.151.62:39983
I20260619 20:28:06.508105 376 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:06.508278 376 heartbeater.cc:511] Master 127.21.151.62:39983 requested a full tablet report, sending...
I20260619 20:28:06.508626 32380 ts_manager.cc:194] Registered new tserver with Master: 151e657b669044f1b6590fb9807b2e26 (127.21.151.3:43545)
I20260619 20:28:06.509087 32380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:49441
I20260619 20:28:06.515180 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:28:06.522547 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.516571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.21.151.62:39983)
0619 20:28:06.516639 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:06.516641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:06.516770 (+ 129us) server_negotiation.cc:407] Connection header received
0619 20:28:06.516792 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:06.516792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:06.516802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:06.516865 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:06.517364 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.517864 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.518547 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.518728 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.518789 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.519972 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:06.519973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:06.519973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:28:06.519979 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:06.521594 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:06.521893 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:06.521894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:06.521894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:06.521993 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:06.522151 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:06.522151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:06.522152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:06.522275 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:06.522349 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.522434 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.522489 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:28:06.524137 32380 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:28:06.525630 32380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:28:06.531994 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.527920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.21.151.2:32917)
0619 20:28:06.528305 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:06.528308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:06.528317 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:28:06.528350 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:06.528352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:06.528388 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:06.528680 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:28:06.528757 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.529458 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.530918 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.531536 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.531646 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.531710 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.531864 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.531902 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":85,"threads_started":1}
I20260619 20:28:06.532003 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.527742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:32917 (local address 127.0.0.1:49070)
0619 20:28:06.528037 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:06.528081 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:06.528150 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:06.528541 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:06.528541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:06.528544 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:28:06.528603 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.528604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.529559 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.529560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.530822 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.530825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.531653 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.531654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.531734 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.531748 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:28:06.531847 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 20:28:06.531903 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":42,"thread_start_us":69,"threads_started":1}
I20260619 20:28:06.532161 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.527842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.21.151.1:44817)
0619 20:28:06.528122 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:06.528127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:06.528247 (+ 120us) server_negotiation.cc:407] Connection header received
0619 20:28:06.528314 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:06.528315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:06.528329 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:06.528395 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:28:06.528575 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.529344 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.530864 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.531649 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.531714 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.531764 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.532046 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.532086 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":85,"threads_started":1}
I20260619 20:28:06.532330 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.527742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:44817 (local address 127.0.0.1:35096)
0619 20:28:06.528169 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:06.528178 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:06.528245 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:06.528421 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:06.528424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:06.528426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:28:06.528501 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.528502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.529445 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.529446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.530785 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.530787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.531795 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.531796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.531860 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.531863 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:28:06.531907 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:28:06.531946 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":35,"thread_start_us":46,"threads_started":1}
I20260619 20:28:06.532804 32641 tablet_service.cc:1467] Processing CreateTablet for tablet ce6f68c3fd8b44f586dd679d17b26299 (DEFAULT_TABLE table=test-table [id=10b6bf58fdc74c819b01e660f11b0aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:28:06.532989 32506 tablet_service.cc:1467] Processing CreateTablet for tablet ce6f68c3fd8b44f586dd679d17b26299 (DEFAULT_TABLE table=test-table [id=10b6bf58fdc74c819b01e660f11b0aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:28:06.533272 32641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6f68c3fd8b44f586dd679d17b26299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:28:06.533362 32506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6f68c3fd8b44f586dd679d17b26299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:28:06.537062 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.527929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:43545 (local address 127.0.0.1:57208)
0619 20:28:06.529592 (+ 1663us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:06.529602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:06.529652 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:06.529825 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:06.529826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:06.529828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:28:06.529873 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.529874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.531097 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.531098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.532169 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.532172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.536835 (+ 4663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.536836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.536902 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.536906 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:28:06.536957 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 20:28:06.536996 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1641,"thread_start_us":69,"threads_started":1}
I20260619 20:28:06.537163 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.528040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.21.151.3:43545)
0619 20:28:06.528202 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:06.528207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:06.529675 (+ 1468us) server_negotiation.cc:407] Connection header received
0619 20:28:06.529720 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:06.529723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:06.529739 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:06.529819 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:28:06.529932 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.530986 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.532273 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.536681 (+ 4408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.536752 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.536800 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.536998 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.537066 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260619 20:28:06.537832 308 tablet_service.cc:1467] Processing CreateTablet for tablet ce6f68c3fd8b44f586dd679d17b26299 (DEFAULT_TABLE table=test-table [id=10b6bf58fdc74c819b01e660f11b0aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:28:06.538153 308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6f68c3fd8b44f586dd679d17b26299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:28:06.539558 393 tablet_bootstrap.cc:492] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82: Bootstrap starting.
I20260619 20:28:06.540271 393 tablet_bootstrap.cc:654] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82: Neither blocks nor log segments found. Creating new log.
I20260619 20:28:06.540824 393 log.cc:824] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82: Log is configured to *not* fsync() on all Append() calls
I20260619 20:28:06.541714 393 tablet_bootstrap.cc:492] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82: No bootstrap required, opened a new log
I20260619 20:28:06.541779 393 ts_tablet_manager.cc:1397] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:28:06.543641 393 raft_consensus.cc:348] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 } } peers { permanent_uuid: "2b61e329a4144494b26c5b29a06d023e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 32917 } } peers { permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 } }
I20260619 20:28:06.543823 393 raft_consensus.cc:374] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:28:06.543861 393 raft_consensus.cc:729] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdffb7805344fb0b6702947bc60bb82, State: Initialized, Role: FOLLOWER
I20260619 20:28:06.543992 394 tablet_bootstrap.cc:492] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e: Bootstrap starting.
I20260619 20:28:06.544014 393 consensus_queue.cc:260] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 } } peers { permanent_uuid: "2b61e329a4144494b26c5b29a06d023e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 32917 } } peers { permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 } }
I20260619 20:28:06.544283 393 ts_tablet_manager.cc:1428] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:28:06.544404 32572 heartbeater.cc:503] Master 127.21.151.62:39983 was elected leader, sending a full tablet report...
I20260619 20:28:06.544651 394 tablet_bootstrap.cc:654] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e: Neither blocks nor log segments found. Creating new log.
I20260619 20:28:06.545200 394 log.cc:824] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:28:06.545542 396 tablet_bootstrap.cc:492] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: Bootstrap starting.
I20260619 20:28:06.546182 396 tablet_bootstrap.cc:654] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: Neither blocks nor log segments found. Creating new log.
I20260619 20:28:06.546310 394 tablet_bootstrap.cc:492] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e: No bootstrap required, opened a new log
I20260619 20:28:06.546387 394 ts_tablet_manager.cc:1397] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:28:06.546710 396 log.cc:824] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: Log is configured to *not* fsync() on all Append() calls
I20260619 20:28:06.547824 396 tablet_bootstrap.cc:492] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: No bootstrap required, opened a new log
I20260619 20:28:06.547894 396 ts_tablet_manager.cc:1397] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:28:06.548583 394 raft_consensus.cc:348] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 } } peers { permanent_uuid: "2b61e329a4144494b26c5b29a06d023e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 32917 } } peers { permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 } }
I20260619 20:28:06.548756 394 raft_consensus.cc:374] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:28:06.548794 394 raft_consensus.cc:729] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b61e329a4144494b26c5b29a06d023e, State: Initialized, Role: FOLLOWER
I20260619 20:28:06.548892 394 consensus_queue.cc:260] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 } } peers { permanent_uuid: "2b61e329a4144494b26c5b29a06d023e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 32917 } } peers { permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 } }
I20260619 20:28:06.549177 394 ts_tablet_manager.cc:1428] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:28:06.549194 32707 heartbeater.cc:503] Master 127.21.151.62:39983 was elected leader, sending a full tablet report...
I20260619 20:28:06.549881 396 raft_consensus.cc:348] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 } } peers { permanent_uuid: "2b61e329a4144494b26c5b29a06d023e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 32917 } } peers { permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 } }
I20260619 20:28:06.550079 396 raft_consensus.cc:374] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:28:06.550110 396 raft_consensus.cc:729] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151e657b669044f1b6590fb9807b2e26, State: Initialized, Role: FOLLOWER
I20260619 20:28:06.550240 396 consensus_queue.cc:260] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 } } peers { permanent_uuid: "2b61e329a4144494b26c5b29a06d023e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 32917 } } peers { permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 } }
I20260619 20:28:06.550467 396 ts_tablet_manager.cc:1428] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:28:06.550518 376 heartbeater.cc:503] Master 127.21.151.62:39983 was elected leader, sending a full tablet report...
W20260619 20:28:06.592648 32708 tablet.cc:2369] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:28:06.695288 32573 tablet.cc:2369] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:28:06.748310 377 tablet.cc:2369] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:28:06.826952 400 raft_consensus.cc:482] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:28:06.827093 400 raft_consensus.cc:504] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 } } peers { permanent_uuid: "2b61e329a4144494b26c5b29a06d023e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 32917 } } peers { permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 } }
I20260619 20:28:06.827461 400 leader_election.cc:288] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cdffb7805344fb0b6702947bc60bb82 (127.21.151.1:44817), 151e657b669044f1b6590fb9807b2e26 (127.21.151.3:43545)
I20260619 20:28:06.831399 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.827752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:56865 (local address 127.21.151.3:43545)
0619 20:28:06.827806 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:06.827808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:06.827901 (+ 93us) server_negotiation.cc:407] Connection header received
0619 20:28:06.827958 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:06.827959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:06.827969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:06.828020 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:28:06.828223 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.828953 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.830477 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.831042 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.831110 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.831119 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.831288 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.831328 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:28:06.831398 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.827728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:35443 (local address 127.21.151.1:44817)
0619 20:28:06.827771 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:06.827774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:06.827841 (+ 67us) server_negotiation.cc:407] Connection header received
0619 20:28:06.827901 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:06.827902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:06.827914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:06.827986 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:28:06.828223 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.828953 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.830477 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.831029 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.831121 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.831129 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.831288 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.831327 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:28:06.831410 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.827698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:44817 (local address 127.21.151.2:35443)
0619 20:28:06.827765 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:06.827777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:06.827825 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:06.828026 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:06.828027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:06.828029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:28:06.828134 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.828136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.829039 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.829040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.830389 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.830393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.831121 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.831121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.831206 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.831210 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:28:06.831267 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:28:06.831316 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":119}
I20260619 20:28:06.831416 402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.827708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:43545 (local address 127.21.151.2:56865)
0619 20:28:06.827826 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:06.827835 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:06.827892 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:06.828037 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:06.828038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:06.828039 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:28:06.828137 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.828138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.829039 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.829040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.830389 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:06.830393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.831121 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.831121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:06.831206 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.831210 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:28:06.831267 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:28:06.831315 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":111,"thread_start_us":73,"threads_started":1}
I20260619 20:28:06.831781 32526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6f68c3fd8b44f586dd679d17b26299" candidate_uuid: "2b61e329a4144494b26c5b29a06d023e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cdffb7805344fb0b6702947bc60bb82" is_pre_election: true
I20260619 20:28:06.831856 328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6f68c3fd8b44f586dd679d17b26299" candidate_uuid: "2b61e329a4144494b26c5b29a06d023e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "151e657b669044f1b6590fb9807b2e26" is_pre_election: true
I20260619 20:28:06.831948 32526 raft_consensus.cc:2454] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b61e329a4144494b26c5b29a06d023e in term 0.
I20260619 20:28:06.832064 328 raft_consensus.cc:2454] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b61e329a4144494b26c5b29a06d023e in term 0.
I20260619 20:28:06.832211 32594 leader_election.cc:302] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b61e329a4144494b26c5b29a06d023e, 2cdffb7805344fb0b6702947bc60bb82; no voters:
I20260619 20:28:06.832372 400 raft_consensus.cc:2781] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:28:06.832420 400 raft_consensus.cc:482] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:28:06.832435 400 raft_consensus.cc:3037] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:28:06.833088 400 raft_consensus.cc:504] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 } } peers { permanent_uuid: "2b61e329a4144494b26c5b29a06d023e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 32917 } } peers { permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 } }
I20260619 20:28:06.833226 400 leader_election.cc:288] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [CANDIDATE]: Term 1 election: Requested vote from peers 2cdffb7805344fb0b6702947bc60bb82 (127.21.151.1:44817), 151e657b669044f1b6590fb9807b2e26 (127.21.151.3:43545)
I20260619 20:28:06.833386 32526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6f68c3fd8b44f586dd679d17b26299" candidate_uuid: "2b61e329a4144494b26c5b29a06d023e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cdffb7805344fb0b6702947bc60bb82"
I20260619 20:28:06.833397 328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6f68c3fd8b44f586dd679d17b26299" candidate_uuid: "2b61e329a4144494b26c5b29a06d023e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "151e657b669044f1b6590fb9807b2e26"
I20260619 20:28:06.833475 32526 raft_consensus.cc:3037] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:28:06.833470 328 raft_consensus.cc:3037] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:28:06.834177 328 raft_consensus.cc:2454] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b61e329a4144494b26c5b29a06d023e in term 1.
I20260619 20:28:06.834182 32526 raft_consensus.cc:2454] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b61e329a4144494b26c5b29a06d023e in term 1.
I20260619 20:28:06.834342 32593 leader_election.cc:302] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 151e657b669044f1b6590fb9807b2e26, 2b61e329a4144494b26c5b29a06d023e; no voters:
I20260619 20:28:06.834457 400 raft_consensus.cc:2781] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:28:06.834591 400 raft_consensus.cc:686] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 1 LEADER]: Becoming Leader. State: Replica: 2b61e329a4144494b26c5b29a06d023e, State: Running, Role: LEADER
I20260619 20:28:06.834712 400 consensus_queue.cc:237] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 } } peers { permanent_uuid: "2b61e329a4144494b26c5b29a06d023e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 32917 } } peers { permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 } }
I20260619 20:28:06.835544 32380 catalog_manager.cc:5495] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b61e329a4144494b26c5b29a06d023e (127.21.151.2). New cstate: current_term: 1 leader_uuid: "2b61e329a4144494b26c5b29a06d023e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b61e329a4144494b26c5b29a06d023e" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 32917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 } health_report { overall_health: UNKNOWN } } }
I20260619 20:28:06.893584 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.891362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.21.151.2:32917)
0619 20:28:06.891428 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:06.891430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:06.891438 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:28:06.891493 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:06.891494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:06.891511 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:06.891570 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:28:06.891712 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.892179 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.892877 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:06.893058 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:06.893105 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:06.893241 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:28:06.893339 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:28:06.893398 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:06.893432 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 20:28:06.893447 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:28:06.895443 32526 raft_consensus.cc:1261] T ce6f68c3fd8b44f586dd679d17b26299 P 2cdffb7805344fb0b6702947bc60bb82 [term 1 FOLLOWER]: Refusing update from remote peer 2b61e329a4144494b26c5b29a06d023e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:28:06.895452 328 raft_consensus.cc:1261] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26 [term 1 FOLLOWER]: Refusing update from remote peer 2b61e329a4144494b26c5b29a06d023e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:28:06.895859 403 consensus_queue.cc:1035] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cdffb7805344fb0b6702947bc60bb82" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:28:06.896008 400 consensus_queue.cc:1035] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [LEADER]: Connected to new peer: Peer: permanent_uuid: "151e657b669044f1b6590fb9807b2e26" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 43545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:28:06.898222 406 mvcc.cc:205] Tried to move back new op lower bound from 7298666032720977920 to 7298666032475660288. Current Snapshot: MvccSnapshot[applied={T|T < 7298666032720977920}]
I20260619 20:28:06.901041 32379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:28:06.901124 32379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:28:06.901831 32379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd74c52800504b1db735c6544f417432: Sending DeleteTablet for 3 replicas of tablet ce6f68c3fd8b44f586dd679d17b26299
I20260619 20:28:06.902138 32506 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6f68c3fd8b44f586dd679d17b26299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:28:06 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:35096
I20260619 20:28:06.902228 32641 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6f68c3fd8b44f586dd679d17b26299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:28:06 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:49070
I20260619 20:28:06.902362 308 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6f68c3fd8b44f586dd679d17b26299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:28:06 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57208
I20260619 20:28:06.902514 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 32442
I20260619 20:28:06.904285 418 tablet_replica.cc:330] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: stopping tablet replica
I20260619 20:28:06.904399 418 raft_consensus.cc:2229] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:28:06.904510 418 raft_consensus.cc:2258] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:28:06.904973 418 ts_tablet_manager.cc:1905] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:28:06.906996 418 ts_tablet_manager.cc:1918] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:28:06.907073 418 log.cc:1196] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/ce6f68c3fd8b44f586dd679d17b26299
I20260619 20:28:06.907334 418 ts_tablet_manager.cc:1939] T ce6f68c3fd8b44f586dd679d17b26299 P 151e657b669044f1b6590fb9807b2e26: Deleting consensus metadata
I20260619 20:28:06.907757 32364 catalog_manager.cc:4841] TS 151e657b669044f1b6590fb9807b2e26 (127.21.151.3:43545): tablet ce6f68c3fd8b44f586dd679d17b26299 (table test-table [id=10b6bf58fdc74c819b01e660f11b0aad]) successfully deleted
I20260619 20:28:06.907819 417 tablet_replica.cc:330] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e: stopping tablet replica
I20260619 20:28:06.908041 417 raft_consensus.cc:2229] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:28:06.908226 417 raft_consensus.cc:2258] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:28:06.908716 417 ts_tablet_manager.cc:1905] T ce6f68c3fd8b44f586dd679d17b26299 P 2b61e329a4144494b26c5b29a06d023e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:28:06.908896 32365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:44817 (error 108)
I20260619 20:28:06.909409 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.909279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:44817 (local address 127.0.0.1:35102)
0619 20:28:06.909335 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:06.909370 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:44817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 20:28:06.909520 32365 catalog_manager.cc:4584] TS 2cdffb7805344fb0b6702947bc60bb82 (127.21.151.1:44817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce6f68c3fd8b44f586dd679d17b26299: Network error: Client connection negotiation failed: client connection to 127.21.151.1:44817: connect: Connection refused (error 111)
I20260619 20:28:06.909785 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 32577
I20260619 20:28:06.914183 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 32712
I20260619 20:28:06.914239 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:06.914112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:32917 (local address 127.0.0.1:49090)
0619 20:28:06.914173 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:06.914205 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.2:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 20:28:06.918727 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 32348
2026-06-19T20:28:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2208 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:28:06 dist-test-slave-f9rk krb5kdc[422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:28:06 dist-test-slave-f9rk krb5kdc[422](info): set up 2 sockets
Jun 19 20:28:06 dist-test-slave-f9rk krb5kdc[422](info): commencing operation
krb5kdc: starting...
W20260619 20:28:07.969662 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:28:08 dist-test-slave-f9rk krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:28:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:28:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/rangeradmin_127.21.151.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/rangeradmin_127.21.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/rangerlookup_127.21.151.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/rangerlookup_127.21.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/HTTP_127.21.151.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/HTTP_127.21.151.20@KRBTEST.COM.keytab.
I20260619 20:28:08.124264 22108 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskM6qw9r/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/postgres -l logfile start
2026-06-19 20:28:10.398 UTC [453] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 20:28:10.398 UTC [453] LOG: listening on IPv4 address "127.21.151.20", port 42537
2026-06-19 20:28:10.401 UTC [453] LOG: listening on Unix socket "/tmp/.s.PGSQL.42537"
2026-06-19 20:28:10.436 UTC [456] LOG: database system was shut down at 2026-06-19 20:28:09 UTC
2026-06-19 20:28:10.440 UTC [453] LOG: database system is ready to accept connections
I20260619 20:28:11.393471 22108 mini_postgres.cc:96] Postgres bound to 42537
2026-06-19 20:28:11.400 UTC [464] FATAL: database "slave" does not exist
127.21.151.20:42537 - accepting connections
I20260619 20:28:11.400606 22108 mini_ranger.cc:161] Starting Ranger...
I20260619 20:28:11.416129 22108 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 20:28:11.856231 22108 mini_ranger.cc:87] Created ranger Postgres database
I20260619 20:28:11.856320 22108 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-admin
2026-06-19 20:28:12,126 [I] DB FLAVOR :POSTGRES
2026-06-19 20:28:12,127 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:28:12,127 [I] Checking connection..
2026-06-19 20:28:12,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 20:28:12,386 [I] Checking connection passed.
2026-06-19 20:28:12,386 [I] --------- Verifying version history table ---------
2026-06-19 20:28:12,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:28:12,627 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:28:12,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:28:12,863 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:28:12,864 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 20:28:12,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 20:28:13.117 UTC [528] WARNING: there is no transaction in progress
2026-06-19 20:28:13,123 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 20:28:13,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:28:13,385 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 20:28:13,385 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:28:13,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 20:28:13,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 20:28:13,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f9rk.c.gcp-upstream.internal','N') ;"
2026-06-19 20:28:14,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:28:14,396 [I] Table x_portal_user does not exist in database ranger
2026-06-19 20:28:14,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:28:14,640 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 20:28:14,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:28:14,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:28:15,133 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 20:28:15,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 20:28:16.078 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.094 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.109 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.124 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.139 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.159 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.213 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.219 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.233 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.244 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.254 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.269 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.279 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.285 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.292 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16.299 UTC [637] WARNING: there is no transaction in progress
2026-06-19 20:28:16,896 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 20:28:16,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 20:28:17,127 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 20:28:17,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:28:17,381 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 20:28:17,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:28:17,611 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 20:28:17,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:28:17,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:28:18,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f9rk.c.gcp-upstream.internal';"
2026-06-19 20:28:18,351 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 20:28:18,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 20:28:18,588 [I] DB_PATCHES have already been applied
I20260619 20:28:18.594700 22108 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-admin:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskM6qw9r/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 20:28:18.594791 22108 mini_ranger.cc:193] Using host: 127.21.151.20
I20260619 20:28:18.597898 22108 mini_ranger.cc:227] Ranger admin URL: http://127.21.151.20:51079
Jun 19, 2026 8:28:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:28:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:28:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 8:28:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 8:28:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 8:28:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 8:28:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 20:28:19.700600 22108 mini_ranger.cc:160] Time spent starting Ranger: real 8.300s user 0.000s sys 0.009s
Jun 19, 2026 8:28:38 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 8:28:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 8:28:38 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 8:28:38 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 8:28:39 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 20:28:40.283703 22108 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/rangerkms_127.21.151.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/rangerkms_127.21.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.151.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.151.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.151.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/HTTP_127.21.151.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.151.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/HTTP_127.21.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 20:28:40 dist-test-slave-f9rk krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 20:28:40.375083 22108 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 20:28:40.386853 22108 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 20:28:41.244473 22108 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 20:28:41.244566 22108 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-kms
I20260619 20:28:41.244583 22108 mini_ranger_kms.cc:227] Using postgres at 127.21.151.20:42537
2026-06-19 20:28:41,416 [I] DB FLAVOR :POSTGRES
2026-06-19 20:28:41,416 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:28:41,416 [I] Checking connection
2026-06-19 20:28:41,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 20:28:41,684 [I] connection success
2026-06-19 20:28:41,684 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 20:28:41,684 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 20:28:41,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 20:28:41,926 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 20:28:41,926 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:28:41,926 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 20:28:41,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:42537/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 20:28:42,180 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 20:28:42.677745 22108 mini_ranger_kms.cc:310] Created kms service
I20260619 20:28:42.872144 22108 mini_ranger_kms.cc:326] Created kudu user
I20260619 20:28:42.916568 22108 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 20:28:42 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:28:43.324080 22108 mini_ranger_kms.cc:384] Added ranger policy
I20260619 20:28:43.324203 22108 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-kms:/tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 20:28:43.324221 22108 mini_ranger_kms.cc:240] Using host: 127.21.151.20
I20260619 20:28:43.328125 22108 mini_ranger_kms.cc:276] Ranger KMS PID: 818
I20260619 20:28:43.328256 22108 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.151.20:59827
Jun 19, 2026 8:28:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:28:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:28:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 8:28:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 8:28:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 8:28:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 20:28:45 dist-test-slave-f9rk krb5kdc[422](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781900925, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.151.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:28:45.632004 22108 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.257s user 0.000s sys 0.155s
I20260619 20:28:45.632120 22108 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 20:28:45.632229 22108 mini_ranger_kms.cc:401] 127.21.151.20:59827/kms/v1/keys
Jun 19 20:28:45 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900925, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.151.20@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
Jun 19 20:28:46 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900925, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.151.20@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
Jun 19 20:28:48 dist-test-slave-f9rk krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:28:48.592675 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:33443
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:33443
--ranger_config_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:59827/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:28:48.697317 869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:48.697705 869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:48.697759 869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:48.699687 869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:28:48.699746 869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:48.699761 869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:48.699774 869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:28:48.699786 869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:28:48.701489 869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.151.20:59827/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:33443
--ranger_config_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:33443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:48.701745 869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:48.702008 869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:48.707042 875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:48.707050 874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:48.707094 877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:48.707376 869 server_base.cc:1034] running on GCE node
I20260619 20:28:48.707696 869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:48.708050 869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:48.709203 869 hybrid_clock.cc:648] HybridClock initialized: now 1781900928709187 us; error 33 us; skew 500 ppm
Jun 19 20:28:48 dist-test-slave-f9rk krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900928, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:28:48.711834 869 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:28:48.712276 869 webserver.cc:460] Webserver started at http://127.21.151.62:41577/ using document root <none> and password file <none>
I20260619 20:28:48.712507 869 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:48.712569 869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:48.712702 869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:28:48 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900928, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:28:48.774287 869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "cee984d0ec7c407cbaeb8675ab10b849"
format_stamp: "Formatted at 2026-06-19 20:28:48 on dist-test-slave-f9rk"
server_key: "16d3a711a263a3f10ea5d12591d8d643"
server_key_iv: "8ad02a22b5af72a7a9c1c3c0bc59edc0"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:48.774727 869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "cee984d0ec7c407cbaeb8675ab10b849"
format_stamp: "Formatted at 2026-06-19 20:28:48 on dist-test-slave-f9rk"
server_key: "16d3a711a263a3f10ea5d12591d8d643"
server_key_iv: "8ad02a22b5af72a7a9c1c3c0bc59edc0"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:48.776270 869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 20:28:48.791529 884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:48.791824 869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:28:48.791950 869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "cee984d0ec7c407cbaeb8675ab10b849"
format_stamp: "Formatted at 2026-06-19 20:28:48 on dist-test-slave-f9rk"
server_key: "16d3a711a263a3f10ea5d12591d8d643"
server_key_iv: "8ad02a22b5af72a7a9c1c3c0bc59edc0"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:48.792078 869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:48.829586 869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:48.831565 869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:48.831780 869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:48.836720 869 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:33443
I20260619 20:28:48.836745 936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:33443 every 8 connection(s)
I20260619 20:28:48.837148 869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:28:48.837913 937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:28:48.839807 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 869
I20260619 20:28:48.839926 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:28:48.840128 22108 external_mini_cluster.cc:1428] Setting key 3cf98d3b884989db248ffb0fbbf2fc69
I20260619 20:28:48.840581 937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849: Bootstrap starting.
I20260619 20:28:48.841234 937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849: Neither blocks nor log segments found. Creating new log.
I20260619 20:28:48.841543 937 log.cc:824] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849: Log is configured to *not* fsync() on all Append() calls
I20260619 20:28:48.842334 937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849: No bootstrap required, opened a new log
I20260619 20:28:48.844797 937 raft_consensus.cc:348] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee984d0ec7c407cbaeb8675ab10b849" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33443 } }
I20260619 20:28:48.844980 937 raft_consensus.cc:374] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:28:48.845013 937 raft_consensus.cc:729] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee984d0ec7c407cbaeb8675ab10b849, State: Initialized, Role: FOLLOWER
I20260619 20:28:48.845463 937 consensus_queue.cc:260] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee984d0ec7c407cbaeb8675ab10b849" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33443 } }
I20260619 20:28:48.845554 937 raft_consensus.cc:388] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:28:48.845587 937 raft_consensus.cc:482] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:28:48.845659 937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:28:48.846333 937 raft_consensus.cc:504] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee984d0ec7c407cbaeb8675ab10b849" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33443 } }
I20260619 20:28:48.846446 937 leader_election.cc:302] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cee984d0ec7c407cbaeb8675ab10b849; no voters:
I20260619 20:28:48.846632 937 leader_election.cc:288] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:28:48.846688 942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:28:48.846886 942 raft_consensus.cc:686] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [term 1 LEADER]: Becoming Leader. State: Replica: cee984d0ec7c407cbaeb8675ab10b849, State: Running, Role: LEADER
I20260619 20:28:48.846966 937 sys_catalog.cc:564] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 20:28:48 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:28:48.847033 942 consensus_queue.cc:237] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee984d0ec7c407cbaeb8675ab10b849" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33443 } }
I20260619 20:28:48.847450 942 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cee984d0ec7c407cbaeb8675ab10b849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee984d0ec7c407cbaeb8675ab10b849" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33443 } } }
I20260619 20:28:48.847605 942 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [sys.catalog]: This master's current role is: LEADER
I20260619 20:28:48.847820 942 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cee984d0ec7c407cbaeb8675ab10b849. Latest consensus state: current_term: 1 leader_uuid: "cee984d0ec7c407cbaeb8675ab10b849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee984d0ec7c407cbaeb8675ab10b849" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33443 } } }
I20260619 20:28:48.847939 942 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849 [sys.catalog]: This master's current role is: LEADER
I20260619 20:28:48.848067 951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:28:48.848589 951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:28:48.848840 937 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 20:28:48.853294 959 mvcc.cc:205] Tried to move back new op lower bound from 7298666204579143680 to 7298666204558049280. Current Snapshot: MvccSnapshot[applied={T|T < 7298666204579143680}]
I20260619 20:28:48.855297 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:48.840754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.21.151.62:33443)
0619 20:28:48.840949 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:48.840959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:48.840991 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:28:48.841532 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:48.841542 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:48.841604 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:48.841757 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:48.842481 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:48.843374 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:48.844214 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:48.844682 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:48.844786 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:48.847574 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:48.847578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:48.847578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:28:48.847591 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:48.852278 (+ 4687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:48.852817 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:48.852822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:48.852825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:48.853074 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:48.853327 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:48.853328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:48.853329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:48.853774 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:48.853995 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:48.854544 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 20:28:48.854970 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
I20260619 20:28:48.855701 951 catalog_manager.cc:1347] Generated new cluster ID: b02c359edd794f0794af7c465c6c194f
I20260619 20:28:48.855804 951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:28:48.859234 951 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:28:48.859885 951 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:28:48.869863 951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849: Generated new TSK 0
I20260619 20:28:48.870152 951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:28:49.705919 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33443
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:28:49.830606 982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:49.830893 982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:49.830932 982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:49.832700 982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:49.832758 982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:49.832798 982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:28:49.834805 982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.151.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:49.835070 982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:49.835438 982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:49.841868 989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:49.842074 982 server_base.cc:1034] running on GCE node
W20260619 20:28:49.842260 991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:49.842443 988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:49.842679 982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:49.842960 982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:49.844116 982 hybrid_clock.cc:648] HybridClock initialized: now 1781900929844081 us; error 50 us; skew 500 ppm
Jun 19 20:28:49 dist-test-slave-f9rk krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900929, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:28:49.847747 982 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:28:49.848273 982 webserver.cc:460] Webserver started at http://127.21.151.1:34279/ using document root <none> and password file <none>
I20260619 20:28:49.848672 982 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:49.848770 982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:49.848933 982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:28:49 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900929, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:28:49.863003 982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "c6fc40d1505a40cfb6d2a43f58668267"
format_stamp: "Formatted at 2026-06-19 20:28:49 on dist-test-slave-f9rk"
server_key: "5f980a8b000f475e7734bb7c54996803"
server_key_iv: "1f0fbeed4256c62749688fd15da57ac9"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:49.863551 982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "c6fc40d1505a40cfb6d2a43f58668267"
format_stamp: "Formatted at 2026-06-19 20:28:49 on dist-test-slave-f9rk"
server_key: "5f980a8b000f475e7734bb7c54996803"
server_key_iv: "1f0fbeed4256c62749688fd15da57ac9"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:49.865475 982 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260619 20:28:49.884172 1000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:49.884459 982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:28:49.884572 982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "c6fc40d1505a40cfb6d2a43f58668267"
format_stamp: "Formatted at 2026-06-19 20:28:49 on dist-test-slave-f9rk"
server_key: "5f980a8b000f475e7734bb7c54996803"
server_key_iv: "1f0fbeed4256c62749688fd15da57ac9"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:49.884671 982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:49.907806 982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:49.910190 982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:49.910396 982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:49.910663 982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:49.911051 982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:49.911134 982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:49.911222 982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:49.911289 982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:49.918015 982 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:44733
I20260619 20:28:49.918490 982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:28:49.923520 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 982
I20260619 20:28:49.923635 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:28:49.923821 22108 external_mini_cluster.cc:1428] Setting key 75b220a12a256d745d1e91567eb34229
I20260619 20:28:49.920650 1114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:44733 every 8 connection(s)
Jun 19 20:28:49 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900929, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
I20260619 20:28:49.944376 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:49.921833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:35319 (local address 127.21.151.62:33443)
0619 20:28:49.922927 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:49.922930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:49.922937 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:28:49.922962 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:49.922962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:49.922975 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:49.923028 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:49.926127 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:49.926934 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:49.928419 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:49.928721 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:49.928799 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:49.932597 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:49.932599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:49.932600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:28:49.932610 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:49.934420 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:49.934911 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:49.934913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:49.934913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:49.934978 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:49.935287 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:49.935288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:49.935288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:49.935445 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:49.935550 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:49.944150 (+ 8600us) server_negotiation.cc:299] Negotiation successful
0619 20:28:49.944262 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":49,"threads_started":1}
I20260619 20:28:49.944515 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:49.921545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33443 (local address 127.21.151.1:35319)
0619 20:28:49.922096 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:49.922141 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:49.922753 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:49.923151 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:49.923155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:49.925678 (+ 2523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:49.925974 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:49.925984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:49.927055 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:49.927056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:49.928305 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:49.928308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:49.929946 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:49.929947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:49.930101 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:49.930767 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:49.930778 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:49.932469 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:49.934520 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:49.934527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:49.934529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:49.934819 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:49.935046 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:49.935047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:49.935047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:49.935178 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:49.943723 (+ 8545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:49.943734 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:49.943947 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 20:28:49.944082 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 20:28:49.944272 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":78,"threads_started":1}
I20260619 20:28:49.945369 1115 heartbeater.cc:347] Connected to a master server at 127.21.151.62:33443
I20260619 20:28:49.945482 1115 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:49.945668 1115 heartbeater.cc:511] Master 127.21.151.62:33443 requested a full tablet report, sending...
I20260619 20:28:49.947289 901 ts_manager.cc:194] Registered new tserver with Master: c6fc40d1505a40cfb6d2a43f58668267 (127.21.151.1:44733)
I20260619 20:28:49.948264 901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:35319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:28:49.962826 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33443
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:28:50 dist-test-slave-f9rk krb5kdc[422](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781900930, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 20:28:50.090811 1121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:50.091039 1121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:50.091077 1121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:50.092396 1121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:50.092481 1121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:50.092540 1121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:28:50.094410 1121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.151.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:50.094671 1121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:50.094965 1121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:50.100764 1131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:50.101475 1128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:50.101683 1129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:50.102664 1121 server_base.cc:1034] running on GCE node
I20260619 20:28:50.102926 1121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:50.103211 1121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:50.104393 1121 hybrid_clock.cc:648] HybridClock initialized: now 1781900930104369 us; error 38 us; skew 500 ppm
Jun 19 20:28:50 dist-test-slave-f9rk krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900930, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:28:50.107775 1121 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:28:50.108325 1121 webserver.cc:460] Webserver started at http://127.21.151.2:35863/ using document root <none> and password file <none>
I20260619 20:28:50.108629 1121 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:50.108728 1121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:50.108891 1121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:28:50 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900930, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:28:50.122964 1121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "5ba96ca37d184b33bdd3dc7251c46fe2"
format_stamp: "Formatted at 2026-06-19 20:28:50 on dist-test-slave-f9rk"
server_key: "39b9c4379a7764ae2afd578b3d7d0f1a"
server_key_iv: "7fcf697187ddfc700f69a2f8bfa9fe4d"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:50.123376 1121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "5ba96ca37d184b33bdd3dc7251c46fe2"
format_stamp: "Formatted at 2026-06-19 20:28:50 on dist-test-slave-f9rk"
server_key: "39b9c4379a7764ae2afd578b3d7d0f1a"
server_key_iv: "7fcf697187ddfc700f69a2f8bfa9fe4d"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:50.132131 1121 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.000s
I20260619 20:28:50.150063 1138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:50.150315 1121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:28:50.150401 1121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "5ba96ca37d184b33bdd3dc7251c46fe2"
format_stamp: "Formatted at 2026-06-19 20:28:50 on dist-test-slave-f9rk"
server_key: "39b9c4379a7764ae2afd578b3d7d0f1a"
server_key_iv: "7fcf697187ddfc700f69a2f8bfa9fe4d"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:50.150496 1121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:50.192662 1121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:50.195048 1121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:50.195195 1121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:50.195426 1121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:50.195835 1121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:50.195875 1121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:50.195906 1121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:50.195981 1121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:50.203557 1121 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:44265
I20260619 20:28:50.203686 1253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:44265 every 8 connection(s)
I20260619 20:28:50.204140 1121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:28:50 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900930, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:28:50.212915 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 1121
I20260619 20:28:50.213030 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:28:50.213207 22108 external_mini_cluster.cc:1428] Setting key 1393ee1db05d4e8400d77da117572530
I20260619 20:28:50.215551 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.204924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:53743 (local address 127.21.151.62:33443)
0619 20:28:50.204974 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:50.204976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:50.205235 (+ 259us) server_negotiation.cc:407] Connection header received
0619 20:28:50.205909 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:50.205911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:50.205928 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:50.206009 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:50.206714 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.207490 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.209043 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.209357 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.209402 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.212021 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:50.212023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:50.212023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:28:50.212031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:50.213902 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:50.214404 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:50.214406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:50.214406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:50.214463 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:50.214747 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:50.214748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:50.214749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:50.214902 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:50.214949 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:50.215348 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 20:28:50.215439 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:28:50.216084 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.204915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33443 (local address 127.21.151.2:53743)
0619 20:28:50.205076 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:50.205125 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:50.205774 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:50.206088 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:50.206093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:50.206335 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:50.206606 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.206613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.207603 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.207604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.208938 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.208941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.209493 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.209494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.209645 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.210244 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:50.210254 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:50.211908 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:50.213999 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:50.214006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:50.214009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:50.214311 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:50.214536 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:50.214537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:50.214537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:50.214669 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:50.215001 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:50.215007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:50.215176 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 20:28:50.215673 (+ 497us) client_negotiation.cc:240] Negotiation successful
0619 20:28:50.215840 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260619 20:28:50.216670 1254 heartbeater.cc:347] Connected to a master server at 127.21.151.62:33443
I20260619 20:28:50.216827 1254 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:50.217099 1254 heartbeater.cc:511] Master 127.21.151.62:33443 requested a full tablet report, sending...
I20260619 20:28:50.217499 901 ts_manager.cc:194] Registered new tserver with Master: 5ba96ca37d184b33bdd3dc7251c46fe2 (127.21.151.2:44265)
I20260619 20:28:50.218008 901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:53743
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:28:50.243208 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33443
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:28:50.354233 1259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:28:50.354449 1259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:28:50.354471 1259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:28:50.355968 1259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:28:50.356065 1259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:28:50.356123 1259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:28:50.358448 1259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.151.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:28:50.358827 1259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:28:50.359129 1259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:28:50.368574 1265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:28:50.368738 1264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:50.369951 1259 server_base.cc:1034] running on GCE node
W20260619 20:28:50.370272 1267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:28:50.370525 1259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:28:50.370805 1259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:28:50.371956 1259 hybrid_clock.cc:648] HybridClock initialized: now 1781900930371931 us; error 41 us; skew 500 ppm
Jun 19 20:28:50 dist-test-slave-f9rk krb5kdc[422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900930, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:28:50.374655 1259 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:28:50.375203 1259 webserver.cc:460] Webserver started at http://127.21.151.3:35897/ using document root <none> and password file <none>
I20260619 20:28:50.375447 1259 fs_manager.cc:359] Metadata directory not provided
I20260619 20:28:50.375516 1259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:28:50.375677 1259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:28:50 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900930, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:28:50.396263 1259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "6dc4dce8e9704981a8619d3ee54eccf1"
format_stamp: "Formatted at 2026-06-19 20:28:50 on dist-test-slave-f9rk"
server_key: "53f73445abbda1d79389c2ba99a48989"
server_key_iv: "7ef0d36b58a65f5207dcb282c4dcf36b"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:50.396806 1259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "6dc4dce8e9704981a8619d3ee54eccf1"
format_stamp: "Formatted at 2026-06-19 20:28:50 on dist-test-slave-f9rk"
server_key: "53f73445abbda1d79389c2ba99a48989"
server_key_iv: "7ef0d36b58a65f5207dcb282c4dcf36b"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:50.398748 1259 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:28:50.427783 1274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:50.428074 1259 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260619 20:28:50.428189 1259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "6dc4dce8e9704981a8619d3ee54eccf1"
format_stamp: "Formatted at 2026-06-19 20:28:50 on dist-test-slave-f9rk"
server_key: "53f73445abbda1d79389c2ba99a48989"
server_key_iv: "7ef0d36b58a65f5207dcb282c4dcf36b"
server_key_version: "kuduclusterkey@0"
I20260619 20:28:50.428292 1259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:28:50.447026 1259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:28:50.450336 1259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:28:50.450556 1259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:28:50.450824 1259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:28:50.451264 1259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:28:50.451359 1259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:50.451455 1259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:28:50.451524 1259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:28:50.458082 1259 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:46055
I20260619 20:28:50.458549 1259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:28:50.464668 1388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:46055 every 8 connection(s)
I20260619 20:28:50.466230 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 1259
I20260619 20:28:50.466354 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:28:50.466547 22108 external_mini_cluster.cc:1428] Setting key 79dd1e6f81978bfdb9a3e890b38ea3a3
Jun 19 20:28:50 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900930, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:28:50.480417 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.465381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:54651 (local address 127.21.151.62:33443)
0619 20:28:50.465430 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:50.465433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:50.466727 (+ 1294us) server_negotiation.cc:407] Connection header received
0619 20:28:50.468616 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:50.468617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:50.468631 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:50.468664 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:50.469540 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.470322 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.473899 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.474211 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.474255 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.477013 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:50.477014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:50.477014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:28:50.477023 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:50.478754 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:50.479261 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:50.479262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:50.479263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:50.479325 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:50.479677 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:50.479678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:50.479678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:50.479818 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:50.479890 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:50.480237 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 20:28:50.480319 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:28:50.480628 1391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.465386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33443 (local address 127.21.151.3:54651)
0619 20:28:50.466584 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:50.466630 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:50.467290 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:50.468880 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:50.468884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:50.469133 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:28:50.469426 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.469434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.470443 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.470444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.473218 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.473222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.474351 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.474351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.474522 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.475192 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:28:50.475203 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:28:50.476894 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:28:50.478842 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:50.478848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:50.478851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:50.479164 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:50.479402 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:28:50.479403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:28:50.479403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:28:50.479552 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:28:50.479891 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:28:50.479898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:28:50.480071 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 20:28:50.480189 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 20:28:50.480357 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"thread_start_us":82,"threads_started":1}
I20260619 20:28:50.481325 1389 heartbeater.cc:347] Connected to a master server at 127.21.151.62:33443
I20260619 20:28:50.481439 1389 heartbeater.cc:464] Registering TS with master...
I20260619 20:28:50.481637 1389 heartbeater.cc:511] Master 127.21.151.62:33443 requested a full tablet report, sending...
I20260619 20:28:50.482052 901 ts_manager.cc:194] Registered new tserver with Master: 6dc4dce8e9704981a8619d3ee54eccf1 (127.21.151.3:46055)
I20260619 20:28:50.482530 901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:54651
I20260619 20:28:50.489009 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:28:50.498308 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.490666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.21.151.62:33443)
0619 20:28:50.490710 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:50.490713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:50.490832 (+ 119us) server_negotiation.cc:407] Connection header received
0619 20:28:50.490885 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:50.490886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:50.490898 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:50.490948 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:28:50.491660 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.492428 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.493517 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.493774 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.493844 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.495359 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:28:50.495360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:28:50.495361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:28:50.495368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:28:50.497117 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:50.497498 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:50.497499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:50.497499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:50.497550 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:28:50.497742 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:28:50.497743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:28:50.497743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:28:50.497860 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:28:50.497942 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:50.498139 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 20:28:50.498213 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:28:50.500506 901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:28:50.502635 901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:28:50.516795 1405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.505696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.21.151.1:44733)
0619 20:28:50.506101 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:50.506106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:50.506539 (+ 433us) server_negotiation.cc:407] Connection header received
0619 20:28:50.506620 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:50.506622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:50.506640 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:50.506723 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:28:50.512620 (+ 5897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.513778 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.515288 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.516102 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.516177 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.516235 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:50.516653 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 20:28:50.516700 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":58,"threads_started":1}
I20260619 20:28:50.517307 1404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.505297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:44733 (local address 127.0.0.1:48474)
0619 20:28:50.506421 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:50.506461 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:50.506540 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:50.507824 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:50.507828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:50.507830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:28:50.512507 (+ 4677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.512511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.513883 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.513884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.515199 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.515202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.516268 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.516269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.516348 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.516361 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:28:50.517168 (+ 807us) client_negotiation.cc:240] Negotiation successful
0619 20:28:50.517232 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":74,"threads_started":1}
I20260619 20:28:50.517947 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.505499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:46055 (local address 127.0.0.1:50178)
0619 20:28:50.512105 (+ 6606us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:50.512116 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:50.512166 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:50.513878 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:50.513879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:50.513880 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:28:50.513917 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.513918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.515206 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.515207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.516660 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.516663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.517715 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.517716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.517778 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.517782 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:28:50.517844 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:28:50.517882 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6589,"thread_start_us":63,"threads_started":1}
I20260619 20:28:50.518000 1406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.505888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.21.151.3:46055)
0619 20:28:50.506264 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:50.506269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:50.512229 (+ 5960us) server_negotiation.cc:407] Connection header received
0619 20:28:50.512290 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:50.512293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:50.512309 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:50.512393 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:28:50.513994 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.515102 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.516773 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.517522 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.517593 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.517678 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:50.517883 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 20:28:50.517924 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":87,"threads_started":1}
I20260619 20:28:50.518220 1048 tablet_service.cc:1467] Processing CreateTablet for tablet f8952b3871774b248486ece9c2f2915f (DEFAULT_TABLE table=test-table [id=21e221e879564ad99934efa663d0b461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:28:50.518606 1048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8952b3871774b248486ece9c2f2915f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:28:50.519037 1318 tablet_service.cc:1467] Processing CreateTablet for tablet f8952b3871774b248486ece9c2f2915f (DEFAULT_TABLE table=test-table [id=21e221e879564ad99934efa663d0b461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:28:50.519387 1318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8952b3871774b248486ece9c2f2915f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:28:50.523268 1408 tablet_bootstrap.cc:492] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: Bootstrap starting.
I20260619 20:28:50.523413 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.505300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:44265 (local address 127.0.0.1:60384)
0619 20:28:50.511700 (+ 6400us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:50.511713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:50.511774 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:50.512490 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:50.512490 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:50.512492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:28:50.512538 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.512540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.519713 (+ 7173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.519714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.520810 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.520813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.523173 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.523174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.523238 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.523243 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:28:50.523289 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 20:28:50.523327 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6378,"thread_start_us":111,"threads_started":1}
I20260619 20:28:50.523895 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.505389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.21.151.2:44265)
0619 20:28:50.506347 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:50.506351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:50.511842 (+ 5491us) server_negotiation.cc:407] Connection header received
0619 20:28:50.511893 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:50.511896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:50.511913 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:50.511991 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:28:50.512645 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.513777 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.522137 (+ 8360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.522992 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.523067 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.523137 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:50.523765 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 20:28:50.523811 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":67,"threads_started":1}
I20260619 20:28:50.524389 1408 tablet_bootstrap.cc:654] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: Neither blocks nor log segments found. Creating new log.
I20260619 20:28:50.524629 1187 tablet_service.cc:1467] Processing CreateTablet for tablet f8952b3871774b248486ece9c2f2915f (DEFAULT_TABLE table=test-table [id=21e221e879564ad99934efa663d0b461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:28:50.524739 1408 log.cc:824] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: Log is configured to *not* fsync() on all Append() calls
I20260619 20:28:50.524947 1187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8952b3871774b248486ece9c2f2915f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:28:50.525058 1407 tablet_bootstrap.cc:492] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267: Bootstrap starting.
I20260619 20:28:50.526167 1407 tablet_bootstrap.cc:654] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267: Neither blocks nor log segments found. Creating new log.
I20260619 20:28:50.527048 1407 log.cc:824] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267: Log is configured to *not* fsync() on all Append() calls
I20260619 20:28:50.528088 1408 tablet_bootstrap.cc:492] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: No bootstrap required, opened a new log
I20260619 20:28:50.528187 1408 ts_tablet_manager.cc:1397] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 20:28:50.533161 1408 raft_consensus.cc:348] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44265 } } peers { permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 } } peers { permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 } }
I20260619 20:28:50.533345 1408 raft_consensus.cc:374] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:28:50.533402 1408 raft_consensus.cc:729] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc4dce8e9704981a8619d3ee54eccf1, State: Initialized, Role: FOLLOWER
I20260619 20:28:50.533510 1408 consensus_queue.cc:260] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44265 } } peers { permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 } } peers { permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 } }
I20260619 20:28:50.533841 1408 ts_tablet_manager.cc:1428] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260619 20:28:50.533897 1389 heartbeater.cc:503] Master 127.21.151.62:33443 was elected leader, sending a full tablet report...
I20260619 20:28:50.534617 1407 tablet_bootstrap.cc:492] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267: No bootstrap required, opened a new log
I20260619 20:28:50.534713 1407 ts_tablet_manager.cc:1397] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260619 20:28:50.535924 1411 tablet_bootstrap.cc:492] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: Bootstrap starting.
I20260619 20:28:50.538182 1411 tablet_bootstrap.cc:654] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: Neither blocks nor log segments found. Creating new log.
Jun 19 20:28:50 dist-test-slave-f9rk krb5kdc[422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900930, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:28:50.539109 1411 log.cc:824] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: Log is configured to *not* fsync() on all Append() calls
I20260619 20:28:50.540040 1411 tablet_bootstrap.cc:492] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: No bootstrap required, opened a new log
I20260619 20:28:50.540123 1411 ts_tablet_manager.cc:1397] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 20:28:50.540163 1407 raft_consensus.cc:348] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44265 } } peers { permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 } } peers { permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 } }
I20260619 20:28:50.540342 1407 raft_consensus.cc:374] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:28:50.540381 1407 raft_consensus.cc:729] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6fc40d1505a40cfb6d2a43f58668267, State: Initialized, Role: FOLLOWER
I20260619 20:28:50.540519 1407 consensus_queue.cc:260] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44265 } } peers { permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 } } peers { permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 } }
I20260619 20:28:50.540849 1407 ts_tablet_manager.cc:1428] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260619 20:28:50.541028 1115 heartbeater.cc:503] Master 127.21.151.62:33443 was elected leader, sending a full tablet report...
I20260619 20:28:50.543694 1411 raft_consensus.cc:348] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44265 } } peers { permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 } } peers { permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 } }
I20260619 20:28:50.543924 1411 raft_consensus.cc:374] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:28:50.544013 1411 raft_consensus.cc:729] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba96ca37d184b33bdd3dc7251c46fe2, State: Initialized, Role: FOLLOWER
I20260619 20:28:50.544207 1411 consensus_queue.cc:260] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44265 } } peers { permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 } } peers { permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 } }
I20260619 20:28:50.544623 1411 ts_tablet_manager.cc:1428] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 20:28:50.544695 1254 heartbeater.cc:503] Master 127.21.151.62:33443 was elected leader, sending a full tablet report...
I20260619 20:28:50.564855 1415 raft_consensus.cc:482] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:28:50.565079 1415 raft_consensus.cc:504] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44265 } } peers { permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 } } peers { permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 } }
I20260619 20:28:50.567116 1415 leader_election.cc:288] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6fc40d1505a40cfb6d2a43f58668267 (127.21.151.1:44733), 6dc4dce8e9704981a8619d3ee54eccf1 (127.21.151.3:46055)
I20260619 20:28:50.570628 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.565734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:44733 (local address 127.21.151.2:57229)
0619 20:28:50.565850 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:50.565862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:50.565902 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:50.566169 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:50.566170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:50.566172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:28:50.566307 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.566309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.568103 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.568104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.569320 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.569323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.570314 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.570314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.570453 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.570457 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:28:50.570507 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:28:50.570555 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 20:28:50.571285 1405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.565980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:57229 (local address 127.21.151.1:44733)
0619 20:28:50.566021 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:50.566023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:50.566030 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:28:50.566058 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:50.566059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:50.566070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:50.566125 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:28:50.566405 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.567256 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.569423 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.570200 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.570272 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.570282 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:50.571172 (+ 890us) server_negotiation.cc:299] Negotiation successful
0619 20:28:50.571221 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:28:50.571889 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8952b3871774b248486ece9c2f2915f" candidate_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6fc40d1505a40cfb6d2a43f58668267" is_pre_election: true
I20260619 20:28:50.572057 1066 raft_consensus.cc:2454] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba96ca37d184b33bdd3dc7251c46fe2 in term 0.
I20260619 20:28:50.573364 1141 leader_election.cc:302] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba96ca37d184b33bdd3dc7251c46fe2, c6fc40d1505a40cfb6d2a43f58668267; no voters:
I20260619 20:28:50.575698 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.567807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:46055 (local address 127.21.151.2:52445)
0619 20:28:50.570690 (+ 2883us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:50.570700 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:28:50.570737 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:28:50.572051 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:28:50.572052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:28:50.572054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:28:50.572087 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.572088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.573559 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.573561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.574615 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:28:50.574617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.575471 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.575472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:28:50.575527 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.575530 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:28:50.575580 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:28:50.575632 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2873,"thread_start_us":160,"threads_started":1}
I20260619 20:28:50.576030 1406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.568025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:52445 (local address 127.21.151.3:46055)
0619 20:28:50.568070 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:50.568072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:50.570789 (+ 2717us) server_negotiation.cc:407] Connection header received
0619 20:28:50.570826 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:50.570827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:50.570836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:50.570880 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:28:50.572229 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.572998 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.574712 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.575330 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.575417 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.575426 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:50.575923 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 20:28:50.575966 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:28:50.576663 1415 raft_consensus.cc:2781] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:28:50.576655 1325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8952b3871774b248486ece9c2f2915f" candidate_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" is_pre_election: true
I20260619 20:28:50.576732 1415 raft_consensus.cc:482] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:28:50.576757 1415 raft_consensus.cc:3037] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:28:50.576822 1325 raft_consensus.cc:2454] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba96ca37d184b33bdd3dc7251c46fe2 in term 0.
I20260619 20:28:50.577877 1415 raft_consensus.cc:504] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44265 } } peers { permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 } } peers { permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 } }
I20260619 20:28:50.578023 1415 leader_election.cc:288] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [CANDIDATE]: Term 1 election: Requested vote from peers c6fc40d1505a40cfb6d2a43f58668267 (127.21.151.1:44733), 6dc4dce8e9704981a8619d3ee54eccf1 (127.21.151.3:46055)
I20260619 20:28:50.578203 1066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8952b3871774b248486ece9c2f2915f" candidate_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6fc40d1505a40cfb6d2a43f58668267"
I20260619 20:28:50.578289 1325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8952b3871774b248486ece9c2f2915f" candidate_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc4dce8e9704981a8619d3ee54eccf1"
I20260619 20:28:50.578317 1066 raft_consensus.cc:3037] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:28:50.578367 1325 raft_consensus.cc:3037] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:28:50.579823 1325 raft_consensus.cc:2454] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba96ca37d184b33bdd3dc7251c46fe2 in term 1.
I20260619 20:28:50.579831 1066 raft_consensus.cc:2454] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba96ca37d184b33bdd3dc7251c46fe2 in term 1.
I20260619 20:28:50.580065 1141 leader_election.cc:302] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba96ca37d184b33bdd3dc7251c46fe2, 6dc4dce8e9704981a8619d3ee54eccf1; no voters:
I20260619 20:28:50.580181 1415 raft_consensus.cc:2781] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:28:50.580344 1415 raft_consensus.cc:686] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 5ba96ca37d184b33bdd3dc7251c46fe2, State: Running, Role: LEADER
I20260619 20:28:50.580461 1415 consensus_queue.cc:237] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44265 } } peers { permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 } } peers { permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 } }
I20260619 20:28:50.581461 901 catalog_manager.cc:5495] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ba96ca37d184b33bdd3dc7251c46fe2 (127.21.151.2). New cstate: current_term: 1 leader_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba96ca37d184b33bdd3dc7251c46fe2" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 } health_report { overall_health: UNKNOWN } } }
I20260619 20:28:50.593765 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.589999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.21.151.2:44265)
0619 20:28:50.590056 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:28:50.590059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:28:50.590082 (+ 23us) server_negotiation.cc:407] Connection header received
0619 20:28:50.590145 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:28:50.590146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:28:50.590169 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:28:50.590250 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:28:50.590421 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.591247 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.592426 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:28:50.592833 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:28:50.592927 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:28:50.593109 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:28:50.593247 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:28:50.593318 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:28:50.593546 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 20:28:50.593568 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:28:50.597448 1325 raft_consensus.cc:1261] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1 [term 1 FOLLOWER]: Refusing update from remote peer 5ba96ca37d184b33bdd3dc7251c46fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:28:50.597466 1066 raft_consensus.cc:1261] T f8952b3871774b248486ece9c2f2915f P c6fc40d1505a40cfb6d2a43f58668267 [term 1 FOLLOWER]: Refusing update from remote peer 5ba96ca37d184b33bdd3dc7251c46fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:28:50.597951 1417 consensus_queue.cc:1035] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6fc40d1505a40cfb6d2a43f58668267" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 44733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:28:50.598105 1415 consensus_queue.cc:1035] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc4dce8e9704981a8619d3ee54eccf1" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:28:50.602502 1420 mvcc.cc:205] Tried to move back new op lower bound from 7298666211723816960 to 7298666211658936320. Current Snapshot: MvccSnapshot[applied={T|T < 7298666211723816960 or (T in {7298666211723816960})}]
I20260619 20:28:50.605132 901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:28:50.605228 901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:28:50.605717 1423 mvcc.cc:205] Tried to move back new op lower bound from 7298666211723816960 to 7298666211658936320. Current Snapshot: MvccSnapshot[applied={T|T < 7298666211723816960}]
I20260619 20:28:50.606057 901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cee984d0ec7c407cbaeb8675ab10b849: Sending DeleteTablet for 3 replicas of tablet f8952b3871774b248486ece9c2f2915f
I20260619 20:28:50.606560 1187 tablet_service.cc:1514] Processing DeleteTablet for tablet f8952b3871774b248486ece9c2f2915f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:28:50 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:60384
I20260619 20:28:50.606601 1048 tablet_service.cc:1514] Processing DeleteTablet for tablet f8952b3871774b248486ece9c2f2915f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:28:50 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:48474
I20260619 20:28:50.606838 1318 tablet_service.cc:1514] Processing DeleteTablet for tablet f8952b3871774b248486ece9c2f2915f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:28:50 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:50178
I20260619 20:28:50.607169 1429 tablet_replica.cc:330] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: stopping tablet replica
I20260619 20:28:50.607245 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 982
I20260619 20:28:50.607327 1431 tablet_replica.cc:330] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: stopping tablet replica
I20260619 20:28:50.607443 1431 raft_consensus.cc:2229] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:28:50.607520 1431 raft_consensus.cc:2258] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:28:50.607946 1429 raft_consensus.cc:2229] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:28:50.608114 1429 raft_consensus.cc:2258] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:28:50.612949 1429 ts_tablet_manager.cc:1905] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:28:50.615196 1429 ts_tablet_manager.cc:1918] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:28:50.615335 1429 log.cc:1196] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/f8952b3871774b248486ece9c2f2915f
I20260619 20:28:50.615741 1429 ts_tablet_manager.cc:1939] T f8952b3871774b248486ece9c2f2915f P 5ba96ca37d184b33bdd3dc7251c46fe2: Deleting consensus metadata
I20260619 20:28:50.616245 886 catalog_manager.cc:4841] TS 5ba96ca37d184b33bdd3dc7251c46fe2 (127.21.151.2:44265): tablet f8952b3871774b248486ece9c2f2915f (table test-table [id=21e221e879564ad99934efa663d0b461]) successfully deleted
I20260619 20:28:50.620285 1431 ts_tablet_manager.cc:1905] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:28:50.621994 1431 ts_tablet_manager.cc:1918] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:28:50.622066 1431 log.cc:1196] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/f8952b3871774b248486ece9c2f2915f
I20260619 20:28:50.622370 1431 ts_tablet_manager.cc:1939] T f8952b3871774b248486ece9c2f2915f P 6dc4dce8e9704981a8619d3ee54eccf1: Deleting consensus metadata
I20260619 20:28:50.622958 887 catalog_manager.cc:4841] TS 6dc4dce8e9704981a8619d3ee54eccf1 (127.21.151.3:46055): tablet f8952b3871774b248486ece9c2f2915f (table test-table [id=21e221e879564ad99934efa663d0b461]) successfully deleted
W20260619 20:28:50.623605 887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:44733 (error 108)
I20260619 20:28:50.624554 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:28:50.624443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:44733 (local address 127.0.0.1:48488)
0619 20:28:50.624490 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:28:50.624520 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:44733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 20:28:50.624964 887 catalog_manager.cc:4584] TS c6fc40d1505a40cfb6d2a43f58668267 (127.21.151.1:44733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8952b3871774b248486ece9c2f2915f: Network error: Client connection negotiation failed: client connection to 127.21.151.1:44733: connect: Connection refused (error 111)
I20260619 20:28:50.625217 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 1121
I20260619 20:28:50.630203 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 1259
I20260619 20:28:50.634817 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 869
I20260619 20:28:50.639884 22108 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 20:28:50.974436 22108 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 20:28:50.974571 22108 mini_ranger.cc:66] Stopping Ranger...
I20260619 20:28:51.319592 22108 mini_ranger.cc:68] Stopped Ranger
2026-06-19 20:28:51.319 UTC [453] LOG: received smart shutdown request
2026-06-19 20:28:51.323 UTC [453] LOG: background worker "logical replication launcher" (PID 462) exited with exit code 1
2026-06-19 20:28:51.323 UTC [457] LOG: shutting down
2026-06-19 20:28:51.484 UTC [453] LOG: database system is shut down
2026-06-19T20:28:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44676 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:28:51 dist-test-slave-f9rk krb5kdc[1441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:28:51 dist-test-slave-f9rk krb5kdc[1441](info): set up 2 sockets
Jun 19 20:28:51 dist-test-slave-f9rk krb5kdc[1441](info): commencing operation
krb5kdc: starting...
W20260619 20:28:52.640657 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:28:52 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:28:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:28:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/rangeradmin_127.21.151.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/rangeradmin_127.21.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/rangerlookup_127.21.151.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/rangerlookup_127.21.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/HTTP_127.21.151.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/HTTP_127.21.151.20@KRBTEST.COM.keytab.
I20260619 20:28:52.798019 22108 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskM6qw9r/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/postgres -l logfile start
2026-06-19 20:28:55.183 UTC [1471] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 20:28:55.183 UTC [1471] LOG: listening on IPv4 address "127.21.151.20", port 49735
2026-06-19 20:28:55.186 UTC [1471] LOG: listening on Unix socket "/tmp/.s.PGSQL.49735"
2026-06-19 20:28:55.221 UTC [1474] LOG: database system was shut down at 2026-06-19 20:28:53 UTC
2026-06-19 20:28:55.225 UTC [1471] LOG: database system is ready to accept connections
I20260619 20:28:56.180605 22108 mini_postgres.cc:96] Postgres bound to 49735
2026-06-19 20:28:56.186 UTC [1482] FATAL: database "slave" does not exist
127.21.151.20:49735 - accepting connections
I20260619 20:28:56.187306 22108 mini_ranger.cc:161] Starting Ranger...
I20260619 20:28:56.202061 22108 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 20:28:56.650506 22108 mini_ranger.cc:87] Created ranger Postgres database
I20260619 20:28:56.650599 22108 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-admin
2026-06-19 20:28:56,941 [I] DB FLAVOR :POSTGRES
2026-06-19 20:28:56,942 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:28:56,942 [I] Checking connection..
2026-06-19 20:28:56,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 20:28:57,205 [I] Checking connection passed.
2026-06-19 20:28:57,205 [I] --------- Verifying version history table ---------
2026-06-19 20:28:57,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:28:57,459 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:28:57,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:28:57,698 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 20:28:57,699 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 20:28:57,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 20:28:57.956 UTC [1546] WARNING: there is no transaction in progress
2026-06-19 20:28:57,962 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 20:28:57,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 20:28:58,194 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 20:28:58,195 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:28:58,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 20:28:58,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 20:28:58,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f9rk.c.gcp-upstream.internal','N') ;"
2026-06-19 20:28:58,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:28:59,200 [I] Table x_portal_user does not exist in database ranger
2026-06-19 20:28:59,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:28:59,444 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 20:28:59,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:28:59,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:28:59,937 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 20:28:59,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 20:29:00.865 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:00.881 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:00.896 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:00.910 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:00.928 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:00.944 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:00.997 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:01.004 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:01.017 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:01.027 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:01.037 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:01.050 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:01.060 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:01.067 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:01.074 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:01.080 UTC [1656] WARNING: there is no transaction in progress
2026-06-19 20:29:01,677 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 20:29:01,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 20:29:01,918 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 20:29:01,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 20:29:02,169 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 20:29:02,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 20:29:02,407 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 20:29:02,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 20:29:02,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 20:29:02,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f9rk.c.gcp-upstream.internal';"
2026-06-19 20:29:03,168 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 20:29:03,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 20:29:03,408 [I] DB_PATCHES have already been applied
I20260619 20:29:03.414126 22108 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-admin:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskM6qw9r/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 20:29:03.414194 22108 mini_ranger.cc:193] Using host: 127.21.151.20
I20260619 20:29:03.417382 22108 mini_ranger.cc:227] Ranger admin URL: http://127.21.151.20:37063
Jun 19, 2026 8:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 8:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 8:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 8:29:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 8:29:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 20:29:04.521530 22108 mini_ranger.cc:160] Time spent starting Ranger: real 8.334s user 0.000s sys 0.009s
Jun 19, 2026 8:29:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 8:29:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 8:29:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 8:29:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 8:29:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 20:29:25.818991 22108 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.151.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.151.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/rangerkms_127.21.151.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.151.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/rangerkms_127.21.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.151.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.151.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.151.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/HTTP_127.21.151.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.151.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/HTTP_127.21.151.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 20:29:25 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900965, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 20:29:25.903367 22108 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 20:29:25.913895 22108 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 20:29:26.717031 22108 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 20:29:26.717123 22108 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-kms
I20260619 20:29:26.717144 22108 mini_ranger_kms.cc:227] Using postgres at 127.21.151.20:49735
2026-06-19 20:29:26,880 [I] DB FLAVOR :POSTGRES
2026-06-19 20:29:26,880 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 20:29:26,880 [I] Checking connection
2026-06-19 20:29:26,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 20:29:27,120 [I] connection success
2026-06-19 20:29:27,120 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 20:29:27,120 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 20:29:27,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 20:29:27,361 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 20:29:27,362 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 20:29:27,362 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 20:29:27,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.151.20:49735/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 20:29:27,615 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 20:29:28.084611 22108 mini_ranger_kms.cc:310] Created kms service
I20260619 20:29:28.295414 22108 mini_ranger_kms.cc:326] Created kudu user
I20260619 20:29:28.343597 22108 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 20:29:28 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900965, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:28.769196 22108 mini_ranger_kms.cc:384] Added ranger policy
I20260619 20:29:28.769315 22108 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-kms:/tmp/dist-test-taskM6qw9r/build/release/bin/postgresql.jar:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskM6qw9r/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 20:29:28.769340 22108 mini_ranger_kms.cc:240] Using host: 127.21.151.20
I20260619 20:29:28.773487 22108 mini_ranger_kms.cc:276] Ranger KMS PID: 1837
I20260619 20:29:28.773586 22108 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.151.20:43011
Jun 19, 2026 8:29:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:29:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:29:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 8:29:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 8:29:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 8:29:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 20:29:31.072341 22108 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.169s user 0.005s sys 0.142s
I20260619 20:29:31.072445 22108 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 20:29:31.072499 22108 mini_ranger_kms.cc:401] 127.21.151.20:43011/kms/v1/keys
Jun 19 20:29:31 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781900971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.151.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 20:29:31 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.151.20@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
Jun 19 20:29:32 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.151.20@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
Jun 19 20:29:34 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:29:34.320595 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44533
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:44533
--ranger_config_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:43011/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:34.412222 1888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:34.412448 1888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:34.412472 1888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:34.413583 1888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:29:34.413617 1888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:34.413631 1888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:34.413657 1888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:29:34.413669 1888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:29:34.415126 1888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.151.20:43011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:44533
--ranger_config_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:34.415302 1888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:34.415469 1888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:34.420421 1893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:34.420436 1894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:34.420631 1888 server_base.cc:1034] running on GCE node
W20260619 20:29:34.420656 1896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:34.420928 1888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:34.421226 1888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:34.422372 1888 hybrid_clock.cc:648] HybridClock initialized: now 1781900974422357 us; error 28 us; skew 500 ppm
Jun 19 20:29:34 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900974, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:34.428561 1888 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:29:34.429016 1888 webserver.cc:460] Webserver started at http://127.21.151.62:40687/ using document root <none> and password file <none>
I20260619 20:29:34.429205 1888 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:34.429242 1888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:34.429347 1888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:29:34 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900974, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:34.492225 1888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "98a491ee9cce4334945778e53403a582"
format_stamp: "Formatted at 2026-06-19 20:29:34 on dist-test-slave-f9rk"
server_key: "8b8eeb9cddb5667ea44b7c31c6d36aec"
server_key_iv: "214a9d48218dfb47a1906b68f8ea5c93"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:34.492686 1888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "98a491ee9cce4334945778e53403a582"
format_stamp: "Formatted at 2026-06-19 20:29:34 on dist-test-slave-f9rk"
server_key: "8b8eeb9cddb5667ea44b7c31c6d36aec"
server_key_iv: "214a9d48218dfb47a1906b68f8ea5c93"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:34.494479 1888 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 20:29:34.507903 1903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:34.508145 1888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:29:34.508244 1888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "98a491ee9cce4334945778e53403a582"
format_stamp: "Formatted at 2026-06-19 20:29:34 on dist-test-slave-f9rk"
server_key: "8b8eeb9cddb5667ea44b7c31c6d36aec"
server_key_iv: "214a9d48218dfb47a1906b68f8ea5c93"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:34.508348 1888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:34.530640 1888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:34.532760 1888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:34.532960 1888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:34.537880 1888 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:44533
I20260619 20:29:34.537912 1956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:44533 every 8 connection(s)
I20260619 20:29:34.538296 1888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:29:34.538991 1957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:34.541424 1957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: Bootstrap starting.
I20260619 20:29:34.542028 1957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:34.542338 1957 log.cc:824] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:34.543016 1957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: No bootstrap required, opened a new log
I20260619 20:29:34.545256 1957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } }
I20260619 20:29:34.545416 1957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:34.545440 1957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a491ee9cce4334945778e53403a582, State: Initialized, Role: FOLLOWER
I20260619 20:29:34.545594 1957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } }
I20260619 20:29:34.545672 1957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:29:34.545714 1957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:29:34.545761 1957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:34.546339 1957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } }
I20260619 20:29:34.546452 1957 leader_election.cc:302] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98a491ee9cce4334945778e53403a582; no voters:
I20260619 20:29:34.546612 1957 leader_election.cc:288] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:29:34.546669 1960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:29:34.546895 1957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:29:34.546880 1960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 1 LEADER]: Becoming Leader. State: Replica: 98a491ee9cce4334945778e53403a582, State: Running, Role: LEADER
I20260619 20:29:34.547079 1960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } }
I20260619 20:29:34.547525 1961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98a491ee9cce4334945778e53403a582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } } }
I20260619 20:29:34.547539 1962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a491ee9cce4334945778e53403a582. Latest consensus state: current_term: 1 leader_uuid: "98a491ee9cce4334945778e53403a582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } } }
I20260619 20:29:34.547683 1961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:34.547714 1962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:34.547775 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 1888
I20260619 20:29:34.547880 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:29:34.547876 1968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:29:34.548086 22108 external_mini_cluster.cc:1428] Setting key a1a4c1b6f79f4c548e61561becf940c6
I20260619 20:29:34.548507 1968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:29:34.549408 1957 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 20:29:34.553946 1968 catalog_manager.cc:1347] Generated new cluster ID: 58be1f252ef34af48e8df0bb952d2608
I20260619 20:29:34.554103 1968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:29:34 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:34.563380 1968 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:29:34.564199 1968 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:29:34.566633 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:34.548581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.21.151.62:44533)
0619 20:29:34.549758 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:34.549779 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:34.549813 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:29:34.551495 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:34.551560 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:34.552087 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:34.552314 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:34.553246 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:34.556559 (+ 3313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:34.557491 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:34.558260 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:34.558381 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:34.561690 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:34.561695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:34.561696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:34.561711 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:34.564216 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:34.564733 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:34.564736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:34.564736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:34.564869 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:34.565101 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:34.565102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:34.565102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:34.565424 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:34.566090 (+ 666us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:34.566172 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 20:29:34.566332 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130,"thread_start_us":81,"threads_started":1}
I20260619 20:29:34.579034 1968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: Generated new TSK 0
I20260619 20:29:34.579264 1968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:29:35.290850 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:43011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44533
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:35.402230 2002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:35.402519 2002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:35.402557 2002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:35.404296 2002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:35.404361 2002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:35.404399 2002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:29:35.406636 2002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.151.20:43011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:35.406898 2002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:35.407200 2002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:35.413872 2008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:35.414222 2010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:35.415093 2002 server_base.cc:1034] running on GCE node
W20260619 20:29:35.415766 2007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:35.416066 2002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:35.416350 2002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:35.432169 2002 hybrid_clock.cc:648] HybridClock initialized: now 1781900975432122 us; error 51 us; skew 500 ppm
Jun 19 20:29:35 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900975, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:35.435873 2002 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:29:35.436498 2002 webserver.cc:460] Webserver started at http://127.21.151.1:38383/ using document root <none> and password file <none>
I20260619 20:29:35.436779 2002 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:35.436854 2002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:35.437026 2002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:29:35 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900975, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:35.452838 2002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "82aa9286db234f1b950c2c6fce304431"
format_stamp: "Formatted at 2026-06-19 20:29:35 on dist-test-slave-f9rk"
server_key: "31ddbec79d2365111cd59fd62428c204"
server_key_iv: "bce3de34385feef8778bca82a9a62f08"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:35.453289 2002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "82aa9286db234f1b950c2c6fce304431"
format_stamp: "Formatted at 2026-06-19 20:29:35 on dist-test-slave-f9rk"
server_key: "31ddbec79d2365111cd59fd62428c204"
server_key_iv: "bce3de34385feef8778bca82a9a62f08"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:35.456599 2002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 20:29:35.473399 2019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:35.473690 2002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 20:29:35.473793 2002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "82aa9286db234f1b950c2c6fce304431"
format_stamp: "Formatted at 2026-06-19 20:29:35 on dist-test-slave-f9rk"
server_key: "31ddbec79d2365111cd59fd62428c204"
server_key_iv: "bce3de34385feef8778bca82a9a62f08"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:35.473896 2002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:35.492930 2002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:35.495365 2002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:35.495553 2002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:35.495918 2002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:35.496352 2002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:35.496418 2002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:35.496466 2002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:35.496502 2002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:35.505481 2002 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:46165
I20260619 20:29:35.506036 2002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:29:35.506634 2133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:46165 every 8 connection(s)
I20260619 20:29:35.507836 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 2002
I20260619 20:29:35.507952 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:29:35.508165 22108 external_mini_cluster.cc:1428] Setting key 1bf794edb7094f3b36ffb5fc0e02e82e
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Jun 19 20:29:35 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900975, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:35.532905 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:35.506880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44533 (local address 127.21.151.1:38889)
0619 20:29:35.507267 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:35.507315 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:35.508031 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:35.508541 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:35.508550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:35.508820 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:35.509242 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:35.509250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:35.510273 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:35.510274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:35.511705 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:35.511708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:35.512338 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:35.512339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:35.512495 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:35.513136 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:35.513147 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:35.527997 (+ 14850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:35.530707 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:35.530722 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:35.530725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:35.531137 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:35.531451 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:35.531452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:35.531452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:35.531669 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:35.532067 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:35.532076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:35.532297 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 20:29:35.532420 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 20:29:35.532618 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":84,"threads_started":1}
I20260619 20:29:35.533696 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:35.507271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:38889 (local address 127.21.151.62:44533)
0619 20:29:35.507447 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:35.507449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:35.507465 (+ 16us) server_negotiation.cc:407] Connection header received
0619 20:29:35.508213 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:35.508215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:35.508231 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:35.508292 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:35.509360 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:35.510163 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:35.511809 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:35.512161 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:35.512257 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:35.528171 (+ 15914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:35.528174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:35.528174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:35.528187 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:35.530552 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:35.531269 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:35.531271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:35.531271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:35.531351 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:35.531777 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:35.531778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:35.531778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:35.531959 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:35.533418 (+ 1459us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:35.533474 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 20:29:35.533581 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260619 20:29:35.534400 2134 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44533
I20260619 20:29:35.534526 2134 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:35.534765 2134 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
I20260619 20:29:35.535460 1921 ts_manager.cc:194] Registered new tserver with Master: 82aa9286db234f1b950c2c6fce304431 (127.21.151.1:46165)
I20260619 20:29:35.536533 1921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:38889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:29:35.558012 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:43011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44533
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:29:35 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781900975, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 20:29:35.676723 2140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:35.677089 2140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:35.677179 2140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:35.678897 2140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:35.679011 2140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:35.679108 2140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:29:35.681334 2140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.151.20:43011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:35.681785 2140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:35.682112 2140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:35.688305 2148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:35.688328 2150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:35.688676 2147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:35.689340 2140 server_base.cc:1034] running on GCE node
I20260619 20:29:35.689563 2140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:35.689843 2140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:35.691012 2140 hybrid_clock.cc:648] HybridClock initialized: now 1781900975690993 us; error 42 us; skew 500 ppm
Jun 19 20:29:35 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900975, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:35.694402 2140 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:29:35.694943 2140 webserver.cc:460] Webserver started at http://127.21.151.2:45011/ using document root <none> and password file <none>
I20260619 20:29:35.695261 2140 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:35.695333 2140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:35.695497 2140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:29:35 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900975, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:35.709285 2140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "f745df5140f7472e8dc134dbdd288960"
format_stamp: "Formatted at 2026-06-19 20:29:35 on dist-test-slave-f9rk"
server_key: "b69c1160745f935ee28ec011bf9d02e8"
server_key_iv: "fa052b4947867466015fe834f1a982ef"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:35.709843 2140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "f745df5140f7472e8dc134dbdd288960"
format_stamp: "Formatted at 2026-06-19 20:29:35 on dist-test-slave-f9rk"
server_key: "b69c1160745f935ee28ec011bf9d02e8"
server_key_iv: "fa052b4947867466015fe834f1a982ef"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:35.711941 2140 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 20:29:35.724757 2157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:35.725047 2140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:29:35.725153 2140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "f745df5140f7472e8dc134dbdd288960"
format_stamp: "Formatted at 2026-06-19 20:29:35 on dist-test-slave-f9rk"
server_key: "b69c1160745f935ee28ec011bf9d02e8"
server_key_iv: "fa052b4947867466015fe834f1a982ef"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:35.725247 2140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:35.764760 2140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:35.767408 2140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:35.767665 2140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:35.768031 2140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:35.768451 2140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:35.768546 2140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:35.768620 2140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:35.768697 2140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:35.775763 2140 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:45135
I20260619 20:29:35.776309 2272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:45135 every 8 connection(s)
I20260619 20:29:35.776371 2140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:29:35.779240 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 2140
I20260619 20:29:35.779340 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:29:35.779520 22108 external_mini_cluster.cc:1428] Setting key 9cb63b4a5e75b974c8a4ea3b95b728c2
Jun 19 20:29:35 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900975, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:35.794777 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:35.777704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:36571 (local address 127.21.151.62:44533)
0619 20:29:35.777770 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:35.777773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:35.783948 (+ 6175us) server_negotiation.cc:407] Connection header received
0619 20:29:35.784815 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:35.784816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:35.784833 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:35.784864 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:35.785733 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:35.786522 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:35.788073 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:35.788411 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:35.788490 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:35.791298 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:35.791300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:35.791300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:35.791309 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:35.793100 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:35.793608 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:35.793609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:35.793609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:35.793675 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:35.794005 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:35.794006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:35.794006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:35.794153 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:35.794227 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:35.794581 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 20:29:35.794677 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:29:35.794965 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:35.777414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44533 (local address 127.21.151.2:36571)
0619 20:29:35.783775 (+ 6361us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:35.783839 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:35.784549 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:35.785073 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:35.785078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:35.785337 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:35.785626 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:35.785632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:35.786644 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:35.786645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:35.787971 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:35.787974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:35.788539 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:35.788540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:35.788718 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:35.789432 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:35.789443 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:35.791192 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:35.793197 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:35.793204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:35.793207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:35.793518 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:35.793757 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:35.793758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:35.793758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:35.793911 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:35.794237 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:35.794244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:35.794415 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 20:29:35.794544 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 20:29:35.794711 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6301,"thread_start_us":94,"threads_started":1}
I20260619 20:29:35.795823 2273 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44533
I20260619 20:29:35.795995 2273 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:35.796242 2273 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
I20260619 20:29:35.796669 1921 ts_manager.cc:194] Registered new tserver with Master: f745df5140f7472e8dc134dbdd288960 (127.21.151.2:45135)
I20260619 20:29:35.797174 1921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:36571
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:29:35.824865 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:43011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44533
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:35.960274 2278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:35.960704 2278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:35.960816 2278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:35.962807 2278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:35.962942 2278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:35.963033 2278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:29:35.965502 2278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.151.20:43011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:35.966037 2278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:35.966401 2278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:35.974169 2286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:35.974213 2283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:35.974509 2284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:35.974861 2278 server_base.cc:1034] running on GCE node
I20260619 20:29:35.975078 2278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:35.975342 2278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:35.979715 2278 hybrid_clock.cc:648] HybridClock initialized: now 1781900975979694 us; error 35 us; skew 500 ppm
Jun 19 20:29:35 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900975, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:35.992321 2278 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:29:35.995757 2278 webserver.cc:460] Webserver started at http://127.21.151.3:37533/ using document root <none> and password file <none>
I20260619 20:29:35.996162 2278 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:35.996295 2278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:35.996481 2278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 20:29:35 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900975, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:36.020783 2278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "76c772775a5e4d2a87706d0feb78b436"
format_stamp: "Formatted at 2026-06-19 20:29:36 on dist-test-slave-f9rk"
server_key: "b0c03d0b03e1ad1b8177a80f80080490"
server_key_iv: "751a05c9fbd5406d000c9a6b4ce47a08"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:36.022459 2278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "76c772775a5e4d2a87706d0feb78b436"
format_stamp: "Formatted at 2026-06-19 20:29:36 on dist-test-slave-f9rk"
server_key: "b0c03d0b03e1ad1b8177a80f80080490"
server_key_iv: "751a05c9fbd5406d000c9a6b4ce47a08"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:36.026278 2278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 20:29:36.042352 2293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.042608 2278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:29:36.042693 2278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "76c772775a5e4d2a87706d0feb78b436"
format_stamp: "Formatted at 2026-06-19 20:29:36 on dist-test-slave-f9rk"
server_key: "b0c03d0b03e1ad1b8177a80f80080490"
server_key_iv: "751a05c9fbd5406d000c9a6b4ce47a08"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:36.042781 2278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:36.113741 2278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:36.116299 2278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:36.116546 2278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:36.116817 2278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:36.117215 2278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:36.117292 2278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.117363 2278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:36.117416 2278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.123590 2278 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:34159
I20260619 20:29:36.126497 2278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:29:36.125922 2407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:34159 every 8 connection(s)
I20260619 20:29:36.134832 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 2278
I20260619 20:29:36.134953 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:29:36.135147 22108 external_mini_cluster.cc:1428] Setting key 9aea172129cb8731ab5d8225aa222eba
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900975, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:36.146222 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:36.128179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:56737 (local address 127.21.151.62:44533)
0619 20:29:36.128243 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:36.128246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:36.136034 (+ 7788us) server_negotiation.cc:407] Connection header received
0619 20:29:36.136780 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:36.136781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:36.136793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:36.136823 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:36.137710 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.138388 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.139895 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.140189 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.140231 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:36.142967 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:36.142968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:36.142968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:36.142977 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:36.144702 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:36.145182 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:36.145183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:36.145184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:36.145241 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:36.145495 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:36.145496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:36.145496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:36.145622 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:36.145661 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:36.146030 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 20:29:36.146121 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 20:29:36.147239 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:36.127870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44533 (local address 127.21.151.3:56737)
0619 20:29:36.135777 (+ 7907us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:36.135941 (+ 164us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:36.136658 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:36.137038 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:36.137044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:36.137307 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:36.137603 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:36.137610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.138505 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.138505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:36.139801 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:36.139804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.140336 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.140337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:36.140476 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:36.141126 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:36.141137 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:36.142845 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:36.144795 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:36.144802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:36.144804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:36.145097 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:36.145305 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:36.145306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:36.145306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:36.145428 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:36.145702 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:36.145708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:36.145869 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 20:29:36.146738 (+ 869us) client_negotiation.cc:240] Negotiation successful
0619 20:29:36.146911 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7843,"thread_start_us":79,"threads_started":1}
I20260619 20:29:36.147966 2408 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44533
I20260619 20:29:36.148073 2408 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:36.148288 2408 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
I20260619 20:29:36.148703 1921 ts_manager.cc:194] Registered new tserver with Master: 76c772775a5e4d2a87706d0feb78b436 (127.21.151.3:34159)
I20260619 20:29:36.149243 1921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:56737
I20260619 20:29:36.158463 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:29:36.159159 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 2002
I20260619 20:29:36.164465 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 2140
I20260619 20:29:36.171159 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 2278
I20260619 20:29:36.177258 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 1888
I20260619 20:29:36.182617 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44533
--webserver_interface=127.21.151.62
--webserver_port=40687
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:44533
--ranger_config_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:43011/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.151.20:43011/kms with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:36.266520 2411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:36.266736 2411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:36.266758 2411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:36.267970 2411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:29:36.268015 2411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:36.268029 2411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:36.268042 2411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:29:36.268105 2411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:29:36.269640 2411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.151.20:43011/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:44533
--ranger_config_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=40687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:36.269888 2411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:36.270135 2411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:36.274881 2416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:36.274863 2419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:36.274865 2417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:36.275065 2411 server_base.cc:1034] running on GCE node
I20260619 20:29:36.275381 2411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:36.275731 2411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:36.276971 2411 hybrid_clock.cc:648] HybridClock initialized: now 1781900976276952 us; error 29 us; skew 500 ppm
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:36.279596 2411 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:29:36.280025 2411 webserver.cc:460] Webserver started at http://127.21.151.62:40687/ using document root <none> and password file <none>
I20260619 20:29:36.280200 2411 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:36.280247 2411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:36.355302 2411 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:29:36.356158 2427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.356339 2411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:29:36.356444 2411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "98a491ee9cce4334945778e53403a582"
format_stamp: "Formatted at 2026-06-19 20:29:34 on dist-test-slave-f9rk"
server_key: "8b8eeb9cddb5667ea44b7c31c6d36aec"
server_key_iv: "214a9d48218dfb47a1906b68f8ea5c93"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:36.356828 2411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:36.369504 2411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:36.371133 2411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:36.371300 2411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:36.375540 2411 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:44533
I20260619 20:29:36.375579 2480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:44533 every 8 connection(s)
I20260619 20:29:36.375989 2411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:29:36.377194 2481 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:29:36.377868 2481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: Bootstrap starting.
I20260619 20:29:36.378559 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 2411
I20260619 20:29:36.378862 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:43011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:46165
--local_ip_for_outbound_sockets=127.21.151.1
--tserver_master_addrs=127.21.151.62:44533
--webserver_port=38383
--webserver_interface=127.21.151.1
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.151.20:43011/kms with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:29:36.386801 2481 log.cc:824] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:36.388669 2481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:29:36.388990 2481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: Bootstrap complete.
I20260619 20:29:36.391360 2481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } }
I20260619 20:29:36.391597 2481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a491ee9cce4334945778e53403a582, State: Initialized, Role: FOLLOWER
I20260619 20:29:36.391860 2481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } }
I20260619 20:29:36.391958 2481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:29:36.392012 2481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:29:36.392060 2481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:29:36.393167 2481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } }
I20260619 20:29:36.393289 2481 leader_election.cc:302] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98a491ee9cce4334945778e53403a582; no voters:
I20260619 20:29:36.393484 2481 leader_election.cc:288] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:29:36.393520 2485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:29:36.393648 2485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [term 2 LEADER]: Becoming Leader. State: Replica: 98a491ee9cce4334945778e53403a582, State: Running, Role: LEADER
I20260619 20:29:36.393756 2481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:29:36.393736 2485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } }
I20260619 20:29:36.394027 2485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a491ee9cce4334945778e53403a582. Latest consensus state: current_term: 2 leader_uuid: "98a491ee9cce4334945778e53403a582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } } }
I20260619 20:29:36.394098 2485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:36.394241 2486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "98a491ee9cce4334945778e53403a582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a491ee9cce4334945778e53403a582" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44533 } } }
I20260619 20:29:36.394332 2486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:36.394420 2489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:29:36.394906 2489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:29:36.395027 2489 catalog_manager.cc:1259] Loaded cluster ID: 58be1f252ef34af48e8df0bb952d2608
I20260619 20:29:36.395061 2489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:29:36.395553 2481 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 20:29:36.398699 2489 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:29:36.399369 2489 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: Loaded TSK: 0
I20260619 20:29:36.399712 2489 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:29:36.470568 2483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:36.470793 2483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:36.470842 2483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:36.472755 2483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:36.472906 2483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:36.472997 2483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:29:36.475261 2483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.151.20:43011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:46165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=38383
--tserver_master_addrs=127.21.151.62:44533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:36.475752 2483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:36.476102 2483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:36.482309 2519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:36.482425 2521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:36.482565 2518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:36.483493 2483 server_base.cc:1034] running on GCE node
I20260619 20:29:36.483731 2483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:36.483999 2483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:36.491410 2483 hybrid_clock.cc:648] HybridClock initialized: now 1781900976491369 us; error 50 us; skew 500 ppm
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:36.494882 2483 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:29:36.495580 2483 webserver.cc:460] Webserver started at http://127.21.151.1:38383/ using document root <none> and password file <none>
I20260619 20:29:36.496047 2483 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:36.496136 2483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:36.530675 2483 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260619 20:29:36.531891 2529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.534736 2483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260619 20:29:36.534888 2483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "82aa9286db234f1b950c2c6fce304431"
format_stamp: "Formatted at 2026-06-19 20:29:35 on dist-test-slave-f9rk"
server_key: "31ddbec79d2365111cd59fd62428c204"
server_key_iv: "bce3de34385feef8778bca82a9a62f08"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:36.535462 2483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:36.568248 2483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:36.570770 2483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:36.570955 2483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:36.571213 2483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:36.571676 2483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:36.571736 2483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.571843 2483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:36.571882 2483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.580569 2483 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:46165
I20260619 20:29:36.581123 2483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:29:36.582057 2642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:46165 every 8 connection(s)
I20260619 20:29:36.585436 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 2483
I20260619 20:29:36.585743 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:43011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:45135
--local_ip_for_outbound_sockets=127.21.151.2
--tserver_master_addrs=127.21.151.62:44533
--webserver_port=45011
--webserver_interface=127.21.151.2
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.151.20:43011/kms with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:36.597922 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:36.584123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:51317 (local address 127.21.151.62:44533)
0619 20:29:36.584310 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:36.584318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:36.584360 (+ 42us) server_negotiation.cc:407] Connection header received
0619 20:29:36.585142 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:36.585148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:36.585193 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:36.585360 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:36.586220 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.587743 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.589166 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.589717 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.589826 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:36.592556 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:36.592562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:36.592562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:36.592576 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:36.595805 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:36.596352 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:36.596356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:36.596356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:36.596471 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:36.596747 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:36.596748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:36.596748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:36.597055 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:36.597148 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:36.597553 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 20:29:36.597686 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260619 20:29:36.597939 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:36.584011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44533 (local address 127.21.151.1:51317)
0619 20:29:36.584190 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:36.584258 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:36.584896 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:36.585528 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:36.585533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:36.585783 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:36.586084 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:36.586091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.587876 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.587877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:36.589075 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:36.589078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.589947 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.589949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:36.590088 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:36.590713 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:36.590724 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:36.592357 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:36.595909 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:36.595917 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:36.595919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:36.596240 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:36.596546 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:36.596546 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:36.596547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:36.596677 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:36.597180 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:36.597186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:36.597355 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 20:29:36.597499 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 20:29:36.597686 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":112,"threads_started":1}
I20260619 20:29:36.598770 2643 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44533
I20260619 20:29:36.598878 2643 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:36.599082 2643 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
W20260619 20:29:36.600000 2643 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:44533 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 20:29:36.600075 2643 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:36.600137 2643 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
I20260619 20:29:36.600592 2643 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:36.600665 2643 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
W20260619 20:29:36.601183 2643 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:29:36.698017 2647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:36.698308 2647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:36.698349 2647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:36.700141 2647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:36.700212 2647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:36.700254 2647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:29:36.702267 2647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.151.20:43011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:45135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=45011
--tserver_master_addrs=127.21.151.62:44533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:36.702508 2647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:36.702780 2647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:36.709143 2653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:36.709220 2655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:36.709148 2652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:36.709780 2647 server_base.cc:1034] running on GCE node
I20260619 20:29:36.710007 2647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:36.710261 2647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:36.711491 2647 hybrid_clock.cc:648] HybridClock initialized: now 1781900976711461 us; error 46 us; skew 500 ppm
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:36.714794 2647 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:29:36.715301 2647 webserver.cc:460] Webserver started at http://127.21.151.2:45011/ using document root <none> and password file <none>
I20260619 20:29:36.715605 2647 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:36.715756 2647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:36.729055 2647 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260619 20:29:36.730050 2663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.730260 2647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:29:36.730360 2647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "f745df5140f7472e8dc134dbdd288960"
format_stamp: "Formatted at 2026-06-19 20:29:35 on dist-test-slave-f9rk"
server_key: "b69c1160745f935ee28ec011bf9d02e8"
server_key_iv: "fa052b4947867466015fe834f1a982ef"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:36.730752 2647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:36.743287 2647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:36.745572 2647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:36.745739 2647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:36.746029 2647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:36.746389 2647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:36.746443 2647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.746506 2647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:36.746533 2647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.753937 2647 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:45135
I20260619 20:29:36.754065 2776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:45135 every 8 connection(s)
I20260619 20:29:36.754426 2647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:36.763849 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 2647
I20260619 20:29:36.764161 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.151.20:43011/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:34159
--local_ip_for_outbound_sockets=127.21.151.3
--tserver_master_addrs=127.21.151.62:44533
--webserver_port=37533
--webserver_interface=127.21.151.3
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.151.20:43011/kms with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:29:36.768165 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:36.755246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:36801 (local address 127.21.151.62:44533)
0619 20:29:36.755321 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:36.755323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:36.755489 (+ 166us) server_negotiation.cc:407] Connection header received
0619 20:29:36.756188 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:36.756190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:36.756204 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:36.756294 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:36.756828 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.757610 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.759171 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.759412 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.759473 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:36.764091 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:36.764093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:36.764093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:36.764105 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:36.766490 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:36.766947 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:36.766948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:36.766949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:36.767019 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:36.767362 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:36.767363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:36.767363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:36.767516 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:36.767614 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:36.767949 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 20:29:36.768055 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 20:29:36.768392 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:36.755206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44533 (local address 127.21.151.2:36801)
0619 20:29:36.755356 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:36.755405 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:36.756077 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:36.756346 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:36.756350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:36.756527 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:36.756717 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:36.756722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.757732 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.757733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:36.759081 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:36.759085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.759532 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.759533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:36.759708 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:36.760773 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:36.760784 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:36.763982 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:36.766575 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:36.766581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:36.766583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:36.766832 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:36.767101 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:36.767102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:36.767102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:36.767276 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:36.767635 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:36.767642 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:36.767804 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 20:29:36.767926 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 20:29:36.768118 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260619 20:29:36.769047 2777 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44533
I20260619 20:29:36.769150 2777 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:36.769349 2777 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
W20260619 20:29:36.770046 2777 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:44533 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 20:29:36.770131 2777 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:36.770205 2777 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
I20260619 20:29:36.770525 2777 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:36.770628 2777 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
W20260619 20:29:36.770924 2777 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:29:36.887466 2780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:36.887765 2780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:36.887804 2780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:36.889493 2780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:36.889554 2780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:36.889593 2780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:29:36.891733 2780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.151.20:43011/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:34159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=37533
--tserver_master_addrs=127.21.151.62:44533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:36.891999 2780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:36.892275 2780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:36.898036 2788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:36.898056 2786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:36.898042 2785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:36.898468 2780 server_base.cc:1034] running on GCE node
I20260619 20:29:36.898615 2780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:36.898861 2780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:36.900135 2780 hybrid_clock.cc:648] HybridClock initialized: now 1781900976900117 us; error 35 us; skew 500 ppm
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:36.903199 2780 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:29:36.903771 2780 webserver.cc:460] Webserver started at http://127.21.151.3:37533/ using document root <none> and password file <none>
I20260619 20:29:36.904074 2780 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:36.904194 2780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:36.921947 2780 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 20:29:36.923076 2797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.923306 2780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:29:36.923410 2780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "76c772775a5e4d2a87706d0feb78b436"
format_stamp: "Formatted at 2026-06-19 20:29:36 on dist-test-slave-f9rk"
server_key: "b0c03d0b03e1ad1b8177a80f80080490"
server_key_iv: "751a05c9fbd5406d000c9a6b4ce47a08"
server_key_version: "kuduclusterkey@0"
I20260619 20:29:36.923893 2780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:36.942250 2780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:36.944461 2780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:36.944639 2780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:36.944933 2780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:36.945257 2780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:36.945312 2780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.945353 2780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:36.945381 2780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:36.952646 2780 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:34159
I20260619 20:29:36.952739 2910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:34159 every 8 connection(s)
I20260619 20:29:36.953140 2780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:29:36.958874 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 2780
Jun 19 20:29:36 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900976, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:36.964077 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:36.953913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:57647 (local address 127.21.151.62:44533)
0619 20:29:36.953961 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:36.953963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:36.954189 (+ 226us) server_negotiation.cc:407] Connection header received
0619 20:29:36.954716 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:36.954717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:36.954729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:36.954787 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:36.955366 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.956126 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.957605 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.957792 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.957852 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:36.960377 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:36.960378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:36.960379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:36.960386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:36.962058 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:36.962563 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:36.962564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:36.962565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:36.962629 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:36.962921 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:36.962923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:36.962923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:36.963066 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:36.963159 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:36.963890 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 20:29:36.963974 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:29:36.964592 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:36.953896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44533 (local address 127.21.151.3:57647)
0619 20:29:36.954063 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:36.954100 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:36.954607 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:36.954870 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:36.954875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:36.955066 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:36.955261 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:36.955265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.956244 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.956245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:36.957512 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:36.957515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.957901 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.957902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:36.958031 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:36.958545 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:36.958556 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:36.960272 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:36.962141 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:36.962148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:36.962150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:36.962467 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:36.962703 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:36.962704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:36.962704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:36.962842 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:36.963188 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:36.963195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:36.963713 (+ 518us) client_negotiation.cc:769] Sending connection context
0619 20:29:36.964175 (+ 462us) client_negotiation.cc:240] Negotiation successful
0619 20:29:36.964353 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260619 20:29:36.965071 2911 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44533
I20260619 20:29:36.965163 2911 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:36.965353 2911 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
W20260619 20:29:36.965971 2911 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:44533 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 20:29:36.966037 2911 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:36.966097 2911 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
I20260619 20:29:36.966392 2911 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:36.966470 2911 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
W20260619 20:29:36.966758 2911 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:29:36.972162 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:36.959729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.21.151.62:44533)
0619 20:29:36.959855 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:36.959857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:36.959864 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:29:36.959884 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:36.959885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:36.959895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:36.959997 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:36.960853 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.961639 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.962935 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:36.963262 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:36.963345 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:36.965039 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:36.965040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:36.965041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:36.965047 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:36.966827 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:36.967231 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:36.967232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:36.967232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:36.967294 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:36.967546 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:36.967547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:36.967548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:36.967711 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:36.967875 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:36.971929 (+ 4054us) server_negotiation.cc:299] Negotiation successful
0619 20:29:36.972035 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":76,"threads_started":1}
Jun 19 20:29:37 dist-test-slave-f9rk krb5kdc[1441](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781900977, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:37.601372 2643 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:37.601547 2643 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
I20260619 20:29:37.601955 2444 ts_manager.cc:194] Registered new tserver with Master: 82aa9286db234f1b950c2c6fce304431 (127.21.151.1:46165)
I20260619 20:29:37.602967 2444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:51317
I20260619 20:29:37.771154 2777 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:37.771342 2777 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
I20260619 20:29:37.771831 2444 ts_manager.cc:194] Registered new tserver with Master: f745df5140f7472e8dc134dbdd288960 (127.21.151.2:45135)
I20260619 20:29:37.772401 2444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:36801
Jun 19 20:29:37 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900977, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:37.966956 2911 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:37.967119 2911 heartbeater.cc:511] Master 127.21.151.62:44533 requested a full tablet report, sending...
I20260619 20:29:37.967481 2444 ts_manager.cc:194] Registered new tserver with Master: 76c772775a5e4d2a87706d0feb78b436 (127.21.151.3:34159)
I20260619 20:29:37.968020 2444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:57647
I20260619 20:29:37.968041 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:29:37.977245 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:37.969285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.21.151.62:44533)
0619 20:29:37.969404 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:37.969406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:37.969415 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:29:37.969482 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:37.969482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:37.969493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:37.969570 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:37.970254 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.971012 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.972110 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.972418 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.972484 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:37.974001 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:37.974002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:37.974003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:37.974011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:37.975856 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:37.976241 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:37.976242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:37.976242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:37.976303 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:37.976532 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:37.976533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:37.976533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:37.976657 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:37.976752 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:37.976924 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 20:29:37.977016 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260619 20:29:37.979763 2444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:29:37.981927 2444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:29:37.989511 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:37.984986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.21.151.3:34159)
0619 20:29:37.985118 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:37.985124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:37.985254 (+ 130us) server_negotiation.cc:407] Connection header received
0619 20:29:37.985343 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:37.985346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:37.985362 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:37.985441 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:37.985693 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.986757 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.988269 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.989060 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.989143 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:37.989202 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:37.989381 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 20:29:37.989425 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":51,"threads_started":1}
I20260619 20:29:37.989513 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:37.984723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:34159 (local address 127.0.0.1:48368)
0619 20:29:37.985134 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:37.985175 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:37.985254 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:37.985541 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:37.985544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:37.985546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:37.985610 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:37.985611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.986851 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.986852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:37.988175 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:37.988178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.989166 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.989167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:37.989253 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:37.989269 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 20:29:37.989358 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 20:29:37.989416 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":76,"threads_started":1}
I20260619 20:29:37.990790 2829 tablet_service.cc:1467] Processing CreateTablet for tablet 646233ea022b476cbc66fd40ab6beee4 (DEFAULT_TABLE table=test-table [id=a651e4f8ee544ef4901005dfdd348a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:29:37.991150 2829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646233ea022b476cbc66fd40ab6beee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:37.993917 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:37.984748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:46165 (local address 127.0.0.1:57638)
0619 20:29:37.989596 (+ 4848us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:37.989606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:37.989705 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:37.989971 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:37.989972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:37.989973 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:37.990003 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:37.990004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.991266 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.991267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:37.992445 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:37.992449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.993689 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.993690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:37.993755 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:37.993758 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:29:37.993798 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 20:29:37.993842 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4838,"thread_start_us":95,"threads_started":1}
I20260619 20:29:37.994194 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:37.984982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.21.151.1:46165)
0619 20:29:37.985401 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:37.985406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:37.989676 (+ 4270us) server_negotiation.cc:407] Connection header received
0619 20:29:37.989788 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:37.989791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:37.989808 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:37.989860 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:37.990084 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.991167 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.992563 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.993391 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.993452 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:37.993521 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:37.994074 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 20:29:37.994121 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":56,"threads_started":1}
I20260619 20:29:37.994843 2569 tablet_service.cc:1467] Processing CreateTablet for tablet 646233ea022b476cbc66fd40ab6beee4 (DEFAULT_TABLE table=test-table [id=a651e4f8ee544ef4901005dfdd348a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:29:37.995177 2569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646233ea022b476cbc66fd40ab6beee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:37.997982 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:37.984730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:45135 (local address 127.0.0.1:59358)
0619 20:29:37.985564 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:37.985574 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:37.985614 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:37.993564 (+ 7950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:37.993566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:37.993568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:37.993606 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:37.993607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.995661 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.995662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:37.996776 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:37.996779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.997766 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.997768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:37.997835 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:37.997840 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:29:37.997872 (+ 32us) client_negotiation.cc:240] Negotiation successful
0619 20:29:37.997909 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":63,"threads_started":1}
I20260619 20:29:37.998536 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:37.984984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.21.151.2:45135)
0619 20:29:37.993113 (+ 8129us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:37.993119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:37.993130 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:29:37.993164 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:37.993166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:37.993183 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:37.993261 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:37.994489 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.995532 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.996880 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:37.997662 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:37.998309 (+ 647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:37.998376 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:37.998402 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 20:29:37.998441 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8086,"thread_start_us":46,"threads_started":1}
I20260619 20:29:37.999431 2711 tablet_service.cc:1467] Processing CreateTablet for tablet 646233ea022b476cbc66fd40ab6beee4 (DEFAULT_TABLE table=test-table [id=a651e4f8ee544ef4901005dfdd348a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:29:37.999837 2711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646233ea022b476cbc66fd40ab6beee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:38.003291 2945 tablet_bootstrap.cc:492] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: Bootstrap starting.
I20260619 20:29:38.003520 2944 tablet_bootstrap.cc:492] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431: Bootstrap starting.
I20260619 20:29:38.004002 2945 tablet_bootstrap.cc:654] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:38.004276 2944 tablet_bootstrap.cc:654] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:38.004328 2945 log.cc:824] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:38.004688 2944 log.cc:824] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:38.004698 2946 tablet_bootstrap.cc:492] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: Bootstrap starting.
I20260619 20:29:38.005278 2946 tablet_bootstrap.cc:654] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:38.005590 2946 log.cc:824] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:38.005987 2945 tablet_bootstrap.cc:492] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: No bootstrap required, opened a new log
I20260619 20:29:38.006104 2945 ts_tablet_manager.cc:1397] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 20:29:38.006302 2946 tablet_bootstrap.cc:492] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: No bootstrap required, opened a new log
I20260619 20:29:38.006373 2946 ts_tablet_manager.cc:1397] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:29:38.008806 2945 raft_consensus.cc:348] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c772775a5e4d2a87706d0feb78b436" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34159 } } peers { permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 } } peers { permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 } }
I20260619 20:29:38.008962 2946 raft_consensus.cc:348] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c772775a5e4d2a87706d0feb78b436" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34159 } } peers { permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 } } peers { permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 } }
I20260619 20:29:38.009073 2945 raft_consensus.cc:374] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:38.009131 2946 raft_consensus.cc:374] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:38.009186 2946 raft_consensus.cc:729] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f745df5140f7472e8dc134dbdd288960, State: Initialized, Role: FOLLOWER
I20260619 20:29:38.009174 2945 raft_consensus.cc:729] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c772775a5e4d2a87706d0feb78b436, State: Initialized, Role: FOLLOWER
I20260619 20:29:38.009362 2946 consensus_queue.cc:260] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c772775a5e4d2a87706d0feb78b436" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34159 } } peers { permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 } } peers { permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 } }
I20260619 20:29:38.009454 2945 consensus_queue.cc:260] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c772775a5e4d2a87706d0feb78b436" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34159 } } peers { permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 } } peers { permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 } }
I20260619 20:29:38.009649 2946 ts_tablet_manager.cc:1428] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 20:29:38.009778 2777 heartbeater.cc:503] Master 127.21.151.62:44533 was elected leader, sending a full tablet report...
I20260619 20:29:38.009898 2945 ts_tablet_manager.cc:1428] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260619 20:29:38.010038 2911 heartbeater.cc:503] Master 127.21.151.62:44533 was elected leader, sending a full tablet report...
I20260619 20:29:38.011888 2944 tablet_bootstrap.cc:492] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431: No bootstrap required, opened a new log
I20260619 20:29:38.011960 2944 ts_tablet_manager.cc:1397] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260619 20:29:38.013870 2944 raft_consensus.cc:348] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c772775a5e4d2a87706d0feb78b436" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34159 } } peers { permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 } } peers { permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 } }
I20260619 20:29:38.014029 2944 raft_consensus.cc:374] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:38.014073 2944 raft_consensus.cc:729] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82aa9286db234f1b950c2c6fce304431, State: Initialized, Role: FOLLOWER
I20260619 20:29:38.014196 2944 consensus_queue.cc:260] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c772775a5e4d2a87706d0feb78b436" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34159 } } peers { permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 } } peers { permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 } }
I20260619 20:29:38.014530 2643 heartbeater.cc:503] Master 127.21.151.62:44533 was elected leader, sending a full tablet report...
I20260619 20:29:38.014532 2944 ts_tablet_manager.cc:1428] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260619 20:29:38.083596 2644 tablet.cc:2369] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 20:29:38 dist-test-slave-f9rk krb5kdc[1441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900977, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for HTTP/127.21.151.20@KRBTEST.COM
I20260619 20:29:38.163185 2951 raft_consensus.cc:482] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:29:38.163362 2951 raft_consensus.cc:504] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c772775a5e4d2a87706d0feb78b436" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34159 } } peers { permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 } } peers { permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 } }
I20260619 20:29:38.163851 2951 leader_election.cc:288] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82aa9286db234f1b950c2c6fce304431 (127.21.151.1:46165), f745df5140f7472e8dc134dbdd288960 (127.21.151.2:45135)
I20260619 20:29:38.168963 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:38.164604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:57629 (local address 127.21.151.1:46165)
0619 20:29:38.164683 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:38.164686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:38.164848 (+ 162us) server_negotiation.cc:407] Connection header received
0619 20:29:38.164970 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:38.164972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:38.164985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:38.165022 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:38.165271 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:38.166042 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:38.167566 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:38.168087 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:38.168133 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:38.168141 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:38.168845 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 20:29:38.168891 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 20:29:38.169203 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:38.164078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:46165 (local address 127.21.151.3:57629)
0619 20:29:38.164770 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:38.164784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:38.164884 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:38.165105 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:38.165106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:38.165108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:38.165194 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:38.165195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:38.166149 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:38.166150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:38.167424 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:38.167427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:38.168696 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:38.168697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:38.168766 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:38.168770 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:29:38.169077 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 20:29:38.169118 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":108,"threads_started":1}
I20260619 20:29:38.169538 2597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646233ea022b476cbc66fd40ab6beee4" candidate_uuid: "76c772775a5e4d2a87706d0feb78b436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82aa9286db234f1b950c2c6fce304431" is_pre_election: true
I20260619 20:29:38.169700 2597 raft_consensus.cc:2454] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c772775a5e4d2a87706d0feb78b436 in term 0.
I20260619 20:29:38.169791 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:38.164686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:39053 (local address 127.21.151.2:45135)
0619 20:29:38.164790 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:38.164792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:38.166364 (+ 1572us) server_negotiation.cc:407] Connection header received
0619 20:29:38.166471 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:38.166472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:38.166482 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:38.166511 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:38.166739 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:38.167459 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:38.168836 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:38.169400 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:38.169465 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:38.169473 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:38.169680 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 20:29:38.169722 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 20:29:38.169916 2799 leader_election.cc:302] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c772775a5e4d2a87706d0feb78b436, 82aa9286db234f1b950c2c6fce304431; no voters:
I20260619 20:29:38.170046 2951 raft_consensus.cc:2781] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:29:38.170118 2951 raft_consensus.cc:482] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:29:38.170145 2951 raft_consensus.cc:3037] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:38.170056 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:38.164325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:45135 (local address 127.21.151.3:39053)
0619 20:29:38.166296 (+ 1971us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:38.166308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:38.166394 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:38.166576 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:38.166577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:38.166578 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:38.166660 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:38.166662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:38.167588 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:38.167589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:38.168733 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:38.168736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:38.169537 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:38.169538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:38.169602 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:38.169604 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:29:38.169918 (+ 314us) client_negotiation.cc:240] Negotiation successful
0619 20:29:38.169966 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1950,"thread_start_us":68,"threads_started":1}
I20260619 20:29:38.170737 2731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646233ea022b476cbc66fd40ab6beee4" candidate_uuid: "76c772775a5e4d2a87706d0feb78b436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f745df5140f7472e8dc134dbdd288960" is_pre_election: true
I20260619 20:29:38.170895 2731 raft_consensus.cc:2454] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76c772775a5e4d2a87706d0feb78b436 in term 0.
I20260619 20:29:38.171026 2951 raft_consensus.cc:504] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c772775a5e4d2a87706d0feb78b436" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34159 } } peers { permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 } } peers { permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 } }
I20260619 20:29:38.171198 2951 leader_election.cc:288] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [CANDIDATE]: Term 1 election: Requested vote from peers 82aa9286db234f1b950c2c6fce304431 (127.21.151.1:46165), f745df5140f7472e8dc134dbdd288960 (127.21.151.2:45135)
I20260619 20:29:38.171315 2731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646233ea022b476cbc66fd40ab6beee4" candidate_uuid: "76c772775a5e4d2a87706d0feb78b436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f745df5140f7472e8dc134dbdd288960"
I20260619 20:29:38.171388 2731 raft_consensus.cc:3037] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:38.171582 2597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646233ea022b476cbc66fd40ab6beee4" candidate_uuid: "76c772775a5e4d2a87706d0feb78b436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82aa9286db234f1b950c2c6fce304431"
I20260619 20:29:38.171687 2597 raft_consensus.cc:3037] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:38.172348 2731 raft_consensus.cc:2454] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c772775a5e4d2a87706d0feb78b436 in term 1.
I20260619 20:29:38.172514 2798 leader_election.cc:302] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76c772775a5e4d2a87706d0feb78b436, f745df5140f7472e8dc134dbdd288960; no voters:
I20260619 20:29:38.172533 2597 raft_consensus.cc:2454] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76c772775a5e4d2a87706d0feb78b436 in term 1.
I20260619 20:29:38.172621 2951 raft_consensus.cc:2781] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:29:38.172773 2951 raft_consensus.cc:686] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 1 LEADER]: Becoming Leader. State: Replica: 76c772775a5e4d2a87706d0feb78b436, State: Running, Role: LEADER
I20260619 20:29:38.172879 2951 consensus_queue.cc:237] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c772775a5e4d2a87706d0feb78b436" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34159 } } peers { permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 } } peers { permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 } }
I20260619 20:29:38.173730 2444 catalog_manager.cc:5495] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76c772775a5e4d2a87706d0feb78b436 (127.21.151.3). New cstate: current_term: 1 leader_uuid: "76c772775a5e4d2a87706d0feb78b436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76c772775a5e4d2a87706d0feb78b436" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 34159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 } health_report { overall_health: UNKNOWN } } }
W20260619 20:29:38.204514 2912 tablet.cc:2369] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:29:38.217231 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:38.213736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.21.151.3:34159)
0619 20:29:38.213784 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:38.213787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:38.213797 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:29:38.213842 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:38.213843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:38.213865 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:38.213978 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:29:38.214134 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:38.214899 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:38.216078 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:38.216367 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:38.216434 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:38.216662 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:29:38.216800 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:29:38.216877 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:38.217126 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 20:29:38.217147 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:29:38.219684 2731 raft_consensus.cc:1261] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960 [term 1 FOLLOWER]: Refusing update from remote peer 76c772775a5e4d2a87706d0feb78b436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:29:38.220109 2951 consensus_queue.cc:1035] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f745df5140f7472e8dc134dbdd288960" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:29:38.223122 2597 raft_consensus.cc:1261] T 646233ea022b476cbc66fd40ab6beee4 P 82aa9286db234f1b950c2c6fce304431 [term 1 FOLLOWER]: Refusing update from remote peer 76c772775a5e4d2a87706d0feb78b436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:29:38.223413 2951 consensus_queue.cc:1035] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82aa9286db234f1b950c2c6fce304431" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:29:38.225888 2961 mvcc.cc:205] Tried to move back new op lower bound from 7298666406784573440 to 7298666406596837376. Current Snapshot: MvccSnapshot[applied={T|T < 7298666406784573440}]
I20260619 20:29:38.230973 2444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:29:38.231094 2444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:29:38.231854 2444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98a491ee9cce4334945778e53403a582: Sending DeleteTablet for 3 replicas of tablet 646233ea022b476cbc66fd40ab6beee4
I20260619 20:29:38.232316 2829 tablet_service.cc:1514] Processing DeleteTablet for tablet 646233ea022b476cbc66fd40ab6beee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:29:38 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:48368
I20260619 20:29:38.232566 2569 tablet_service.cc:1514] Processing DeleteTablet for tablet 646233ea022b476cbc66fd40ab6beee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:29:38 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57638
I20260619 20:29:38.232931 2711 tablet_service.cc:1514] Processing DeleteTablet for tablet 646233ea022b476cbc66fd40ab6beee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:29:38 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:59358
I20260619 20:29:38.233362 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 2483
I20260619 20:29:38.235142 2970 tablet_replica.cc:330] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: stopping tablet replica
I20260619 20:29:38.235262 2970 raft_consensus.cc:2229] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:29:38.235375 2970 raft_consensus.cc:2258] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:29:38.240710 2970 ts_tablet_manager.cc:1905] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:29:38.243767 2968 tablet_replica.cc:330] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: stopping tablet replica
I20260619 20:29:38.243922 2968 raft_consensus.cc:2229] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:29:38.244046 2968 raft_consensus.cc:2258] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:29:38.244594 2968 ts_tablet_manager.cc:1905] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:29:38.247697 2968 ts_tablet_manager.cc:1918] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:29:38.248081 2970 ts_tablet_manager.cc:1918] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 20:29:38.248157 2429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:46165 (error 108)
W20260619 20:29:38.247864 2799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:46165 (error 108)
I20260619 20:29:38.248543 2970 log.cc:1196] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/646233ea022b476cbc66fd40ab6beee4
I20260619 20:29:38.248669 2968 log.cc:1196] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/646233ea022b476cbc66fd40ab6beee4
I20260619 20:29:38.249038 2970 ts_tablet_manager.cc:1939] T 646233ea022b476cbc66fd40ab6beee4 P f745df5140f7472e8dc134dbdd288960: Deleting consensus metadata
I20260619 20:29:38.249361 2968 ts_tablet_manager.cc:1939] T 646233ea022b476cbc66fd40ab6beee4 P 76c772775a5e4d2a87706d0feb78b436: Deleting consensus metadata
I20260619 20:29:38.249874 2430 catalog_manager.cc:4841] TS 76c772775a5e4d2a87706d0feb78b436 (127.21.151.3:34159): tablet 646233ea022b476cbc66fd40ab6beee4 (table test-table [id=a651e4f8ee544ef4901005dfdd348a89]) successfully deleted
I20260619 20:29:38.250525 2428 catalog_manager.cc:4841] TS f745df5140f7472e8dc134dbdd288960 (127.21.151.2:45135): tablet 646233ea022b476cbc66fd40ab6beee4 (table test-table [id=a651e4f8ee544ef4901005dfdd348a89]) successfully deleted
I20260619 20:29:38.256110 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:38.255943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:46165 (local address 127.0.0.1:57648)
0619 20:29:38.256011 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:38.256063 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 20:29:38.256273 2429 catalog_manager.cc:4584] TS 82aa9286db234f1b950c2c6fce304431 (127.21.151.1:46165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 646233ea022b476cbc66fd40ab6beee4: Network error: Client connection negotiation failed: client connection to 127.21.151.1:46165: connect: Connection refused (error 111)
I20260619 20:29:38.256525 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 2647
I20260619 20:29:38.256696 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:38.256494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:46165 (local address 127.21.151.3:56195)
0619 20:29:38.256606 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:38.256656 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 20:29:38.264369 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 2780
I20260619 20:29:38.270854 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 2411
I20260619 20:29:38.276768 22108 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 20:29:38.611261 22108 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 20:29:38.611378 22108 mini_ranger.cc:66] Stopping Ranger...
I20260619 20:29:38.955782 22108 mini_ranger.cc:68] Stopped Ranger
2026-06-19 20:29:38.955 UTC [1471] LOG: received smart shutdown request
2026-06-19 20:29:38.959 UTC [1471] LOG: background worker "logical replication launcher" (PID 1480) exited with exit code 1
2026-06-19 20:29:38.960 UTC [1475] LOG: shutting down
2026-06-19 20:29:39.111 UTC [1471] LOG: database system is shut down
2026-06-19T20:29:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47623 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:29:39 dist-test-slave-f9rk krb5kdc[2981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:29:39 dist-test-slave-f9rk krb5kdc[2981](info): set up 2 sockets
Jun 19 20:29:39 dist-test-slave-f9rk krb5kdc[2981](info): commencing operation
krb5kdc: starting...
W20260619 20:29:40.256592 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:29:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:29:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:29:40.374083 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:39621
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:39621,127.21.151.61:38045,127.21.151.60:42223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:40.454985 2994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:40.455210 2994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:40.455248 2994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:40.456465 2994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:29:40.456529 2994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:40.456557 2994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:40.456588 2994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:29:40.456616 2994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:29:40.458175 2994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:39621,127.21.151.61:38045,127.21.151.60:42223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:40.458447 2994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:40.458670 2994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:29:40.463662 2994 server_base.cc:1034] running on GCE node
W20260619 20:29:40.463546 2999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:40.463568 3000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:40.463549 3002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:40.464071 2994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:40.464395 2994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:40.465554 2994 hybrid_clock.cc:648] HybridClock initialized: now 1781900980465536 us; error 38 us; skew 500 ppm
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:40.472267 2994 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:29:40.472739 2994 webserver.cc:460] Webserver started at http://127.21.151.62:33561/ using document root <none> and password file <none>
I20260619 20:29:40.473001 2994 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:40.473049 2994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:40.473218 2994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:29:40.474294 2994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8"
format_stamp: "Formatted at 2026-06-19 20:29:40 on dist-test-slave-f9rk"
server_key: "2b69a7e9b28e9ef185a69bf5a781b41c"
server_key_iv: "d8276be1df2034543ca2a669d5946d04"
server_key_version: "encryptionkey@0"
I20260619 20:29:40.474668 2994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8"
format_stamp: "Formatted at 2026-06-19 20:29:40 on dist-test-slave-f9rk"
server_key: "2b69a7e9b28e9ef185a69bf5a781b41c"
server_key_iv: "d8276be1df2034543ca2a669d5946d04"
server_key_version: "encryptionkey@0"
I20260619 20:29:40.476053 2994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:29:40.476789 3009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:40.476969 2994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:29:40.477075 2994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8"
format_stamp: "Formatted at 2026-06-19 20:29:40 on dist-test-slave-f9rk"
server_key: "2b69a7e9b28e9ef185a69bf5a781b41c"
server_key_iv: "d8276be1df2034543ca2a669d5946d04"
server_key_version: "encryptionkey@0"
I20260619 20:29:40.477154 2994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:40.513691 2994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:40.515986 2994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:40.516161 2994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:40.520222 2994 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:39621
I20260619 20:29:40.520288 3061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:39621 every 8 connection(s)
I20260619 20:29:40.520730 2994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:29:40.521613 3062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:40.523344 3062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:40.529583 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 2994
I20260619 20:29:40.529670 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:29:40.529877 22108 external_mini_cluster.cc:1428] Setting key 01438dc398a4b4dbaf8cb1df8dab9e36
I20260619 20:29:40.533186 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.524236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.21.151.62:39621)
0619 20:29:40.524357 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.524362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.524401 (+ 39us) server_negotiation.cc:407] Connection header received
0619 20:29:40.524862 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.524863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.524880 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.524956 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.525501 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.526308 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.527204 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.527515 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.527592 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.529911 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.529920 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.529921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.529928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.531535 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.531961 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.531965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.531965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.532046 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.532297 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.532298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.532298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.532507 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.532595 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.532793 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.532940 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260619 20:29:40.533190 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.524080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39621 (local address 127.0.0.1:49294)
0619 20:29:40.524243 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.524280 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:40.524729 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:40.524988 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:40.524990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:40.525247 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:40.525420 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.525424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.526416 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.526417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.527128 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.527130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.527590 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.527590 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.527716 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.528281 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:40.528289 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:40.529811 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:40.531661 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.531668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.531671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.531873 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.532115 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.532116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.532116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.532211 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.532586 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:40.532591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:40.532653 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 20:29:40.532775 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 20:29:40.533003 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":89,"thread_start_us":67,"threads_started":1}
I20260619 20:29:40.534053 3062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:29:40.534408 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.534285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38045 (local address 127.0.0.1:42422)
0619 20:29:40.534333 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.534378 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:29:40.534538 3013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38045: connect: Connection refused (error 111)
I20260619 20:29:40.534926 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.534829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38045 (local address 127.0.0.1:42436)
0619 20:29:40.534879 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.534904 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 20:29:40.535029 3062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.61:38045: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38045: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.21.151.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab.
Entry for principal kudu/127.21.151.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab.
I20260619 20:29:40.556108 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.61:38045
--webserver_interface=127.21.151.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab
--principal=kudu/127.21.151.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:39621,127.21.151.61:38045,127.21.151.60:42223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:29:40.584231 3062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } attempt: 1
I20260619 20:29:40.584753 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.584616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38045 (local address 127.0.0.1:42450)
0619 20:29:40.584683 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.584722 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:29:40.585132 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.585034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38045 (local address 127.0.0.1:42462)
0619 20:29:40.585081 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.585106 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:29:40.585227 3062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.61:38045: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38045: connect: Connection refused (error 111)
W20260619 20:29:40.638499 3069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:40.638760 3069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:40.638797 3069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:40.640002 3069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:29:40.640075 3069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:40.640098 3069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:40.640120 3069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:29:40.640151 3069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:29:40.641695 3069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:39621,127.21.151.61:38045,127.21.151.60:42223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab
--principal=kudu/127.21.151.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.61:38045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.151.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:40.642023 3069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:40.642266 3069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:40.647161 3074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:40.647179 3075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:40.647176 3077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:40.647217 3069 server_base.cc:1034] running on GCE node
I20260619 20:29:40.647663 3069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:40.647969 3069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:40.649130 3069 hybrid_clock.cc:648] HybridClock initialized: now 1781900980649115 us; error 45 us; skew 500 ppm
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:40.651558 3069 init.cc:377] Logged in from keytab as kudu/127.21.151.61@KRBTEST.COM (short username kudu)
I20260619 20:29:40.652060 3069 webserver.cc:460] Webserver started at http://127.21.151.61:44929/ using document root <none> and password file <none>
I20260619 20:29:40.652269 3069 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:40.652311 3069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:40.652400 3069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:29:40.653363 3062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } attempt: 2
I20260619 20:29:40.653403 3069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/data/instance:
uuid: "40361b2548594a9eac98bfd4dbab7794"
format_stamp: "Formatted at 2026-06-19 20:29:40 on dist-test-slave-f9rk"
server_key: "4c6f9e325942ba8743d878e3a0111613"
server_key_iv: "ba77f9bbb02e676d5575ecb13193a2b3"
server_key_version: "encryptionkey@0"
I20260619 20:29:40.653739 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.653619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38045 (local address 127.0.0.1:42464)
0619 20:29:40.653678 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.653713 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:29:40.653857 3069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/wal/instance:
uuid: "40361b2548594a9eac98bfd4dbab7794"
format_stamp: "Formatted at 2026-06-19 20:29:40 on dist-test-slave-f9rk"
server_key: "4c6f9e325942ba8743d878e3a0111613"
server_key_iv: "ba77f9bbb02e676d5575ecb13193a2b3"
server_key_version: "encryptionkey@0"
I20260619 20:29:40.654054 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.653945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38045 (local address 127.0.0.1:42472)
0619 20:29:40.653987 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.654022 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260619 20:29:40.654158 3062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.61:38045: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38045: connect: Connection refused (error 111)
I20260619 20:29:40.655282 3069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:29:40.656105 3084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:40.656304 3069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:29:40.656378 3069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/wal
uuid: "40361b2548594a9eac98bfd4dbab7794"
format_stamp: "Formatted at 2026-06-19 20:29:40 on dist-test-slave-f9rk"
server_key: "4c6f9e325942ba8743d878e3a0111613"
server_key_iv: "ba77f9bbb02e676d5575ecb13193a2b3"
server_key_version: "encryptionkey@0"
I20260619 20:29:40.656471 3069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:40.668118 3069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:40.669947 3069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:40.670118 3069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:40.674360 3069 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.61:38045
I20260619 20:29:40.674431 3136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.61:38045 every 8 connection(s)
I20260619 20:29:40.674750 3069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
I20260619 20:29:40.675606 3137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:40.677189 3137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:29:40.680938 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 3069
I20260619 20:29:40.681041 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-1/wal/instance
I20260619 20:29:40.681262 22108 external_mini_cluster.cc:1428] Setting key 6645b418736890ad69f252c98a3b3c39
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:40.686412 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.677883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.21.151.62:39621)
0619 20:29:40.677942 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.677944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.678142 (+ 198us) server_negotiation.cc:407] Connection header received
0619 20:29:40.678574 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.678575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.678586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.678649 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.679255 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.679807 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.680818 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.681164 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.681243 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.683428 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.683430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.683430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.683438 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.684874 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.685411 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.685412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.685412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.685466 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.685772 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.685772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.685772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.685878 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.685928 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.686276 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.686341 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 20:29:40.686744 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.677824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39621 (local address 127.0.0.1:49300)
0619 20:29:40.678013 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.678055 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:40.678495 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:40.678704 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:40.678707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:40.678952 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:40.679163 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.679167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.679890 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.679891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.680734 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.680736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.681249 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.681249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.681396 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.681955 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:40.681975 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:40.683317 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:40.684979 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.684989 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.684991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.685334 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.685544 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.685545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.685545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.685702 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.685947 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:40.685954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:40.686130 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 20:29:40.686261 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 20:29:40.686445 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260619 20:29:40.687348 3137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.151.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.60@KRBTEST.COM" created.
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:29:40.695483 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.687689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38045 (local address 127.0.0.1:42480)
0619 20:29:40.687891 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.687903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:40.687947 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:40.688128 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:40.688129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:40.688278 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:40.688336 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.688338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.689172 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.689173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.689903 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.689905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.690367 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.690368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.690423 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.690762 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:40.690767 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:40.692616 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:40.694388 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.694388 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.694389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.694550 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.694766 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.694767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.694767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.694823 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.695144 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:40.695145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:40.695184 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 20:29:40.695256 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 20:29:40.695363 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260619 20:29:40.695670 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.687755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.21.151.61:38045)
0619 20:29:40.687932 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.687936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.687953 (+ 17us) server_negotiation.cc:407] Connection header received
0619 20:29:40.688006 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.688008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.688018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.688097 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.688409 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.689083 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.689972 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.690283 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.690369 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.692710 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.692718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.692719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.692733 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.694293 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.694641 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.694645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.694646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.694700 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.694916 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.694917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.694918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.695072 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.695165 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.695286 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.695429 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260619 20:29:40.696448 3137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:29:40.696817 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.696699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44120)
0619 20:29:40.696751 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.696795 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 20:29:40.696971 3085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
I20260619 20:29:40.698628 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.698530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44124)
0619 20:29:40.698576 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.698602 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 20:29:40.698779 3137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:42223: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab.
Entry for principal kudu/127.21.151.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab.
I20260619 20:29:40.706622 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.60:42223
--webserver_interface=127.21.151.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab
--principal=kudu/127.21.151.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:39621,127.21.151.61:38045,127.21.151.60:42223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:29:40.745309 3062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } attempt: 3
I20260619 20:29:40.748005 3137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } attempt: 1
I20260619 20:29:40.748391 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.748274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44128)
0619 20:29:40.748330 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.748358 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:29:40.748749 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.748621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44142)
0619 20:29:40.748680 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.748713 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 20:29:40.748865 3137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:42223: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:29:40.752789 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.745916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.21.151.61:38045)
0619 20:29:40.745983 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.745985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.746003 (+ 18us) server_negotiation.cc:407] Connection header received
0619 20:29:40.746028 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.746029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.746040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.746093 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.746462 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.746954 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.747941 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.748154 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.748218 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.750203 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.750205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.750205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.750213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.751589 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.751992 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.751993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.751993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.752055 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.752260 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.752261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.752262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.752385 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.752459 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.752621 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.752698 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:29:40.752794 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.745792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38045 (local address 127.0.0.1:42482)
0619 20:29:40.745876 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.745899 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:40.745963 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:40.746115 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:40.746116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:40.746308 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:40.746383 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.746384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.747029 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.747030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.747861 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.747863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.748226 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.748228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.748307 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.748903 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:40.748910 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:40.750116 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:40.751738 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.751739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.751739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.751916 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.752132 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.752133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.752133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.752198 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.752461 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:40.752462 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:40.752510 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 20:29:40.752594 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 20:29:40.752698 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 20:29:40.753212 3062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:29:40.753537 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.753444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44144)
0619 20:29:40.753490 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.753518 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:29:40.753859 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.753803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44146)
0619 20:29:40.753824 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.753843 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260619 20:29:40.753957 3062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:42223: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
I20260619 20:29:40.785101 3062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } attempt: 1
I20260619 20:29:40.785635 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.785480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44152)
0619 20:29:40.785563 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.785604 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 20:29:40.786003 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.785919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44164)
0619 20:29:40.785963 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.785988 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:29:40.786128 3062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:42223: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
W20260619 20:29:40.790643 3144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:40.790925 3144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:40.790952 3144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:40.792230 3144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:29:40.792269 3144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:40.792284 3144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:40.792297 3144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:29:40.792341 3144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:29:40.793846 3144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:39621,127.21.151.61:38045,127.21.151.60:42223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab
--principal=kudu/127.21.151.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.60:42223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.151.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:40.794121 3144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:40.794353 3144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:40.799227 3150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:40.799235 3149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:40.799263 3144 server_base.cc:1034] running on GCE node
W20260619 20:29:40.799245 3152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:40.799737 3144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:40.800081 3144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:40.801258 3144 hybrid_clock.cc:648] HybridClock initialized: now 1781900980801244 us; error 25 us; skew 500 ppm
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:40.803716 3144 init.cc:377] Logged in from keytab as kudu/127.21.151.60@KRBTEST.COM (short username kudu)
I20260619 20:29:40.804095 3144 webserver.cc:460] Webserver started at http://127.21.151.60:36067/ using document root <none> and password file <none>
I20260619 20:29:40.804268 3144 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:40.804306 3144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:40.804395 3144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:29:40.805421 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/data/instance:
uuid: "baaee023bd7c4ed398a2eb59dc0dbf77"
format_stamp: "Formatted at 2026-06-19 20:29:40 on dist-test-slave-f9rk"
server_key: "e68f902ec4c1d9f974d9b4cc7002549b"
server_key_iv: "f1476e313fe207424cae5819ea58caf7"
server_key_version: "encryptionkey@0"
I20260619 20:29:40.805725 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/wal/instance:
uuid: "baaee023bd7c4ed398a2eb59dc0dbf77"
format_stamp: "Formatted at 2026-06-19 20:29:40 on dist-test-slave-f9rk"
server_key: "e68f902ec4c1d9f974d9b4cc7002549b"
server_key_iv: "f1476e313fe207424cae5819ea58caf7"
server_key_version: "encryptionkey@0"
I20260619 20:29:40.807001 3144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:29:40.807726 3159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:40.807878 3144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:29:40.807967 3144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/wal
uuid: "baaee023bd7c4ed398a2eb59dc0dbf77"
format_stamp: "Formatted at 2026-06-19 20:29:40 on dist-test-slave-f9rk"
server_key: "e68f902ec4c1d9f974d9b4cc7002549b"
server_key_iv: "f1476e313fe207424cae5819ea58caf7"
server_key_version: "encryptionkey@0"
I20260619 20:29:40.808027 3144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:40.817029 3137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } attempt: 2
I20260619 20:29:40.817579 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.817435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44166)
0619 20:29:40.817500 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.817544 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 20:29:40.818011 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.817878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44182)
0619 20:29:40.817949 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.817985 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 20:29:40.818148 3137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:42223: Network error: Client connection negotiation failed: client connection to 127.21.151.60:42223: connect: Connection refused (error 111)
I20260619 20:29:40.825997 3144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:40.828465 3144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:40.828629 3144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:40.832612 3144 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.60:42223
I20260619 20:29:40.832644 3211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.60:42223 every 8 connection(s)
I20260619 20:29:40.833019 3144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
I20260619 20:29:40.833899 3212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:40.835410 3212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:40.841630 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 3144
I20260619 20:29:40.841712 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781900829246020-22108-0/minicluster-data/master-2/wal/instance
I20260619 20:29:40.841907 22108 external_mini_cluster.cc:1428] Setting key cca5ba04eeebf3d35ef39ee65a287eb1
I20260619 20:29:40.844604 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.836139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.21.151.62:39621)
0619 20:29:40.836205 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.836207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.836439 (+ 232us) server_negotiation.cc:407] Connection header received
0619 20:29:40.836948 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.836949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.836961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.837019 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.837661 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.838195 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.839211 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.839429 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.839509 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.841592 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.841593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.841594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.841602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.843166 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.843596 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.843597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.843597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.843694 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.843960 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.843961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.843961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.844084 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.844155 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.844432 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.844513 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:29:40.844797 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.836108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39621 (local address 127.0.0.1:49312)
0619 20:29:40.836293 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.836328 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:40.836818 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:40.837098 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:40.837102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:40.837367 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:40.837563 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.837567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.838298 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.838299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.839117 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.839119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.839515 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.839515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.839698 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.840150 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:40.840159 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:40.841492 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:40.843265 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.843272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.843274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.843519 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.843771 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.843772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.843772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.843890 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.844158 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:40.844164 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:40.844300 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 20:29:40.844410 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 20:29:40.844560 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260619 20:29:40.845324 3212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:29:40.850617 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.842301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.21.151.60:42223)
0619 20:29:40.842487 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.842493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.842538 (+ 45us) server_negotiation.cc:407] Connection header received
0619 20:29:40.842605 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.842608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.842622 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.842690 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.843285 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.844119 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.844974 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.845294 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.845378 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.847716 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.847725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.847725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.847737 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.849380 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.849754 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.849757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.849757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.849805 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.850007 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.850007 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.850008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.850177 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.850311 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.850447 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.850506 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260619 20:29:40.852618 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.845931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.21.151.61:38045)
0619 20:29:40.845994 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.845996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.846004 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:29:40.846027 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.846028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.846039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.846104 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.846378 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.846945 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.847703 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.847943 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.848008 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.849738 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.849739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.849740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.849748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.851503 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.851863 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.851865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.851865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.851928 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.852108 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.852108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.852109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.852269 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.852350 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.852500 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.852557 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:29:40.852645 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.845596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38045 (local address 127.0.0.1:42498)
0619 20:29:40.845640 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.845651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:40.845690 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:40.846121 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:40.846122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:40.846253 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:40.846296 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.846297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.847033 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.847034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.847581 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.847582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.848013 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.848014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.848090 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.848327 (+ 237us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:40.848332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:40.849651 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:40.851578 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.851578 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.851579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.851790 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.851999 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.851999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.852000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.852047 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.852356 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:40.852357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:40.852397 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 20:29:40.852470 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:29:40.852553 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:29:40.853157 3212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
I20260619 20:29:40.859298 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.851432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.21.151.62:39621)
0619 20:29:40.851472 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.851473 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.851481 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:29:40.851503 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.851503 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.851513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.851565 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.852190 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.852845 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.853915 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.854178 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.854228 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.856529 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.856531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.856531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.856538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.858055 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.858437 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.858439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.858439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.858493 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.858699 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.858700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.858701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.858844 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.859040 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.859081 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.859203 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:29:40.859777 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.853406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44192)
0619 20:29:40.853462 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.853472 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:40.853522 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:40.853698 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:40.853699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:40.853939 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:40.853969 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.853970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.854784 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.854785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.855231 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.855232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.855708 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.855709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.855765 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.856078 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:40.856083 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:40.857216 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:40.858766 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.858767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.858767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.858945 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.859147 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.859148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.859148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.859221 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.859498 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:40.859499 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:40.859542 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 20:29:40.859600 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 20:29:40.859722 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 20:29:40.860141 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.853448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.21.151.60:42223)
0619 20:29:40.853477 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.853479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.853505 (+ 26us) server_negotiation.cc:407] Connection header received
0619 20:29:40.853572 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.853573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.853581 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.853645 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.854043 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.854705 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.855306 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.855591 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.855705 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.857296 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.857297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.857298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.857320 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.858665 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.859029 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.859030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.859031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.859084 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.859289 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.859290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.859290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.859423 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.859946 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.859987 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.860078 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:29:40.861287 3062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } attempt: 2
I20260619 20:29:40.861788 3212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77: Bootstrap starting.
I20260619 20:29:40.862771 3212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:40.863078 3212 log.cc:824] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:40.863904 3212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77: No bootstrap required, opened a new log
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:29:40.865844 3212 raft_consensus.cc:348] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } }
I20260619 20:29:40.866104 3212 raft_consensus.cc:374] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:40.866195 3212 raft_consensus.cc:729] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baaee023bd7c4ed398a2eb59dc0dbf77, State: Initialized, Role: FOLLOWER
I20260619 20:29:40.866369 3212 consensus_queue.cc:260] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } }
I20260619 20:29:40.866729 3212 sys_catalog.cc:564] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:29:40.867177 3218 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } } }
I20260619 20:29:40.867329 3218 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
I20260619 20:29:40.868335 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.860386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.21.151.61:38045)
0619 20:29:40.860430 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.860432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.860440 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:29:40.860461 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.860462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.860473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.860547 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.861033 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.861703 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.862375 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.862643 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.862717 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.865104 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.865105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.865106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.865114 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.866690 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.867071 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.867072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.867073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.867127 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.867313 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.867314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.867314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.867450 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.867693 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.868153 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.868237 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 20:29:40.869324 3229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 20:29:40.870252 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:29:40.870832 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.861562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44204)
0619 20:29:40.861809 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.861819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:40.861853 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:40.863601 (+ 1748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:40.863602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:40.863780 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:40.863820 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.863822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.864731 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.864732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.865449 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.865451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.866213 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.866215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.866274 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.866635 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:40.866640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:40.867961 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:40.869822 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.869823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.869823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.870015 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.870202 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.870202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.870203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.870255 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.870535 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:40.870535 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:40.870592 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 20:29:40.870648 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:29:40.870735 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260619 20:29:40.871109 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.863379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.21.151.60:42223)
0619 20:29:40.863443 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.863445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.863453 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:29:40.863473 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.863474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.863484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.863587 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.863928 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.864640 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.865530 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.866119 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.866210 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.868051 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.868052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.868052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.868059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.869735 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.870091 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.870092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.870092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.870141 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.870316 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.870317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.870317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.870461 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.870924 (+ 463us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.870964 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.871037 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:29:40.869400 3144 master_runner.cc:186] Error getting master registration for 127.21.151.62:39621: OK, instance_id { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" instance_seqno: 1781900980519024 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:29:40.872350 3144 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 20:29:40.872677 3229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:29:40.873649 3062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8: Bootstrap starting.
I20260619 20:29:40.874253 3062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:40.874572 3062 log.cc:824] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:40.875530 3062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8: No bootstrap required, opened a new log
I20260619 20:29:40.877305 3062 raft_consensus.cc:348] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } }
I20260619 20:29:40.877456 3062 raft_consensus.cc:374] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:40.877487 3062 raft_consensus.cc:729] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af44fb6bb2fc4c90adf3d0c06a1444d8, State: Initialized, Role: FOLLOWER
I20260619 20:29:40.877945 3062 consensus_queue.cc:260] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } }
I20260619 20:29:40.879184 3062 sys_catalog.cc:564] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:29:40.882124 3240 sys_catalog.cc:455] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } } }
I20260619 20:29:40.882282 3240 sys_catalog.cc:458] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:29:40.883430 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.872118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.21.151.60:42223)
0619 20:29:40.872271 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.872273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.875277 (+ 3004us) server_negotiation.cc:407] Connection header received
0619 20:29:40.875324 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.875325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.875335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.875410 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.876076 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.876770 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.877723 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.878021 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.878098 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.879608 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.879609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.879609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.879657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.882169 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.882564 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.882566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.882566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.882628 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.882840 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.882840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.882841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.882980 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.883075 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.883261 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.883344 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
I20260619 20:29:40.882032 2994 master_runner.cc:186] Error getting master registration for 127.21.151.61:38045: OK, instance_id { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" instance_seqno: 1781900980673220 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:29:40.885063 2994 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 20:29:40.886018 3251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 20:29:40.886104 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.871792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.21.151.62:39621)
0619 20:29:40.871843 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.871845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.872054 (+ 209us) server_negotiation.cc:407] Connection header received
0619 20:29:40.872146 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.872147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.872157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.872186 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.873005 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.873708 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.874759 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.875027 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.875119 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.879133 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.879134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.879134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.879141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.881347 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.881745 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.881746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.881747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.881814 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.882033 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.882033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.882034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.885275 (+ 3241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.885391 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.885885 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.885958 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 20:29:40.886144 3251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:29:40.887550 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.872097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.21.151.61:38045)
0619 20:29:40.872134 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.872135 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.879813 (+ 7678us) server_negotiation.cc:407] Connection header received
0619 20:29:40.879921 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.879922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.879934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.879961 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.880641 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.881418 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.882414 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.882718 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.882760 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.884589 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.884590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.884591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.884598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.886286 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.886675 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.886676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.886677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.886733 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.886960 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.886960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.886961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.887097 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.887165 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.887378 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.887454 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:29:40.909317 3137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } attempt: 3
Jun 19 20:29:40 dist-test-slave-f9rk krb5kdc[2981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900980, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
I20260619 20:29:40.916921 3139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.909634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:42223 (local address 127.0.0.1:44224)
0619 20:29:40.909686 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:40.909698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:40.909742 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:40.909896 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:40.909897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:40.910063 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:40.910107 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.910108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.910991 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.910992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.911573 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:40.911575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.912010 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.912011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:40.912069 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.912558 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:40.912564 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:40.914134 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:40.915854 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.915855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.915856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.916035 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.916241 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:40.916242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:40.916242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:40.916300 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:40.916584 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:40.916585 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:40.916634 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 20:29:40.916722 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 20:29:40.916821 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:29:40.916921 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:40.909674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.21.151.60:42223)
0619 20:29:40.909734 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:40.909736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:40.909745 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:29:40.909802 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:40.909803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:40.909812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:40.909878 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:40.910190 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.910911 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.911713 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:40.911916 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:40.912010 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:40.914212 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:40.914214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:40.914214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:40.914222 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:40.915760 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.916109 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.916110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.916111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.916166 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:40.916371 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:40.916372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:40.916372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:40.916496 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:40.916591 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:40.916747 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:29:40.916821 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:29:40.918617 3137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794: Bootstrap starting.
I20260619 20:29:40.919287 3137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:40.919581 3137 log.cc:824] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:40.920392 3137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794: No bootstrap required, opened a new log
I20260619 20:29:40.922276 3137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } }
I20260619 20:29:40.922410 3137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:40.922473 3137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40361b2548594a9eac98bfd4dbab7794, State: Initialized, Role: FOLLOWER
I20260619 20:29:40.922605 3137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } }
I20260619 20:29:40.922880 3254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } } }
I20260619 20:29:40.922946 3137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:29:40.923014 3254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:29:40.924805 3265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:29:40.924916 3265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:29:40.926923 3069 master_runner.cc:263] No leader master found from master 40361b2548594a9eac98bfd4dbab7794
I20260619 20:29:40.926964 3069 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:29:40.954809 3218 raft_consensus.cc:482] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:29:40.954953 3218 raft_consensus.cc:504] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } }
I20260619 20:29:40.955168 3218 leader_election.cc:288] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af44fb6bb2fc4c90adf3d0c06a1444d8 (127.21.151.62:39621), 40361b2548594a9eac98bfd4dbab7794 (127.21.151.61:38045)
I20260619 20:29:40.955793 3111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40361b2548594a9eac98bfd4dbab7794" is_pre_election: true
I20260619 20:29:40.955952 3111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baaee023bd7c4ed398a2eb59dc0dbf77 in term 0.
I20260619 20:29:40.956267 3163 leader_election.cc:302] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40361b2548594a9eac98bfd4dbab7794, baaee023bd7c4ed398a2eb59dc0dbf77; no voters:
I20260619 20:29:40.956532 3036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" is_pre_election: true
I20260619 20:29:40.956667 3036 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baaee023bd7c4ed398a2eb59dc0dbf77 in term 0.
I20260619 20:29:40.956945 3218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:29:40.957000 3218 raft_consensus.cc:482] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:29:40.957023 3218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:40.957912 3218 raft_consensus.cc:504] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } }
I20260619 20:29:40.958022 3218 leader_election.cc:288] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [CANDIDATE]: Term 1 election: Requested vote from peers af44fb6bb2fc4c90adf3d0c06a1444d8 (127.21.151.62:39621), 40361b2548594a9eac98bfd4dbab7794 (127.21.151.61:38045)
I20260619 20:29:40.958431 3111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40361b2548594a9eac98bfd4dbab7794"
I20260619 20:29:40.958518 3111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:40.959437 3111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baaee023bd7c4ed398a2eb59dc0dbf77 in term 1.
I20260619 20:29:40.959786 3163 leader_election.cc:302] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40361b2548594a9eac98bfd4dbab7794, baaee023bd7c4ed398a2eb59dc0dbf77; no voters:
I20260619 20:29:40.959918 3036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8"
I20260619 20:29:40.959997 3036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:40.960850 3036 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baaee023bd7c4ed398a2eb59dc0dbf77 in term 1.
I20260619 20:29:40.961110 3218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:29:40.961262 3218 raft_consensus.cc:686] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [term 1 LEADER]: Becoming Leader. State: Replica: baaee023bd7c4ed398a2eb59dc0dbf77, State: Running, Role: LEADER
I20260619 20:29:40.961342 3218 consensus_queue.cc:237] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } }
I20260619 20:29:40.961715 3218 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baaee023bd7c4ed398a2eb59dc0dbf77. Latest consensus state: current_term: 1 leader_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } } }
I20260619 20:29:40.961783 3218 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:40.962064 3269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:29:40.962270 3269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:29:40.965101 3111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [term 1 FOLLOWER]: Refusing update from remote peer baaee023bd7c4ed398a2eb59dc0dbf77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:29:40.965442 3218 consensus_queue.cc:1035] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:29:40.966660 3254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baaee023bd7c4ed398a2eb59dc0dbf77. Latest consensus state: current_term: 1 leader_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } } }
I20260619 20:29:40.966740 3254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:29:40.967173 3036 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [term 1 FOLLOWER]: Refusing update from remote peer baaee023bd7c4ed398a2eb59dc0dbf77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:29:40.967435 3218 consensus_queue.cc:1035] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:29:40.968648 3240 sys_catalog.cc:455] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baaee023bd7c4ed398a2eb59dc0dbf77. Latest consensus state: current_term: 1 leader_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } } }
I20260619 20:29:40.968732 3240 sys_catalog.cc:458] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:29:40.969992 3267 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } } }
I20260619 20:29:40.970085 3267 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:40.970583 3240 sys_catalog.cc:455] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } } }
I20260619 20:29:40.970655 3240 sys_catalog.cc:458] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:29:40.970898 3270 mvcc.cc:205] Tried to move back new op lower bound from 7298666418030948352 to 7298666418018283520. Current Snapshot: MvccSnapshot[applied={T|T < 7298666418030948352}]
I20260619 20:29:40.971040 3273 mvcc.cc:205] Tried to move back new op lower bound from 7298666418030948352 to 7298666418018283520. Current Snapshot: MvccSnapshot[applied={T|T < 7298666418030948352}]
I20260619 20:29:40.971897 3218 sys_catalog.cc:455] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } } }
I20260619 20:29:40.971971 3218 sys_catalog.cc:458] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:40.972386 3254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44fb6bb2fc4c90adf3d0c06a1444d8" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39621 } } peers { permanent_uuid: "40361b2548594a9eac98bfd4dbab7794" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38045 } } peers { permanent_uuid: "baaee023bd7c4ed398a2eb59dc0dbf77" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 42223 } } }
I20260619 20:29:40.972473 3254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:29:40.972903 3269 catalog_manager.cc:1347] Generated new cluster ID: 29b69b688e9f441c9ae948e6f11d2de6
I20260619 20:29:40.972949 3269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:29:40.978616 3271 mvcc.cc:205] Tried to move back new op lower bound from 7298666418030948352 to 7298666418018283520. Current Snapshot: MvccSnapshot[applied={T|T < 7298666418030948352}]
I20260619 20:29:41.006237 3269 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:29:41.007175 3269 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:29:41.015635 3269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baaee023bd7c4ed398a2eb59dc0dbf77: Generated new TSK 0
I20260619 20:29:41.015935 3269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:29:41.886694 3251 catalog_manager.cc:1259] Loaded cluster ID: 29b69b688e9f441c9ae948e6f11d2de6
I20260619 20:29:41.886775 3251 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8: loading cluster ID for follower catalog manager: success
I20260619 20:29:41.887696 3251 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8: acquiring CA information for follower catalog manager: success
I20260619 20:29:41.888044 3251 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af44fb6bb2fc4c90adf3d0c06a1444d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:29:41.925359 3265 catalog_manager.cc:1259] Loaded cluster ID: 29b69b688e9f441c9ae948e6f11d2de6
I20260619 20:29:41.925436 3265 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794: loading cluster ID for follower catalog manager: success
I20260619 20:29:41.926187 3265 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794: acquiring CA information for follower catalog manager: success
I20260619 20:29:41.926433 3265 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 40361b2548594a9eac98bfd4dbab7794: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:29:42.074057 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 2994
I20260619 20:29:42.079187 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 3069
I20260619 20:29:42.084050 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 3144
2026-06-19T20:29:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2864 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:29:42 dist-test-slave-f9rk krb5kdc[3284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:29:42 dist-test-slave-f9rk krb5kdc[3284](info): set up 2 sockets
Jun 19 20:29:42 dist-test-slave-f9rk krb5kdc[3284](info): commencing operation
krb5kdc: starting...
W20260619 20:29:43.124691 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:29:43 dist-test-slave-f9rk krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:29:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:29:43.238654 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:34671
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:34671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:43.319366 3297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:43.319590 3297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:43.319669 3297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:43.320827 3297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:29:43.320865 3297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:43.320879 3297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:43.320892 3297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:29:43.320904 3297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:29:43.322307 3297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:34671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:34671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:43.322505 3297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:43.322680 3297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:43.327433 3305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:43.327533 3297 server_base.cc:1034] running on GCE node
W20260619 20:29:43.327515 3303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:43.327435 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:43.328007 3297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:43.328248 3297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:43.329422 3297 hybrid_clock.cc:648] HybridClock initialized: now 1781900983329411 us; error 35 us; skew 500 ppm
Jun 19 20:29:43 dist-test-slave-f9rk krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900983, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:43.331833 3297 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:29:43.332273 3297 webserver.cc:460] Webserver started at http://127.21.151.62:45553/ using document root <none> and password file <none>
I20260619 20:29:43.332472 3297 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:43.332513 3297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:43.332600 3297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:29:43.333627 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "3589c245539942c9aa69b1373c991a23"
format_stamp: "Formatted at 2026-06-19 20:29:43 on dist-test-slave-f9rk"
server_key: "ab9e75f18a27ad07ff5dfdb958c81f07"
server_key_iv: "74af0044370e247f4a770dbb1aa43b2e"
server_key_version: "encryptionkey@0"
I20260619 20:29:43.333922 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "3589c245539942c9aa69b1373c991a23"
format_stamp: "Formatted at 2026-06-19 20:29:43 on dist-test-slave-f9rk"
server_key: "ab9e75f18a27ad07ff5dfdb958c81f07"
server_key_iv: "74af0044370e247f4a770dbb1aa43b2e"
server_key_version: "encryptionkey@0"
I20260619 20:29:43.335136 3297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 20:29:43.335819 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:43.335979 3297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:29:43.336071 3297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "3589c245539942c9aa69b1373c991a23"
format_stamp: "Formatted at 2026-06-19 20:29:43 on dist-test-slave-f9rk"
server_key: "ab9e75f18a27ad07ff5dfdb958c81f07"
server_key_iv: "74af0044370e247f4a770dbb1aa43b2e"
server_key_version: "encryptionkey@0"
I20260619 20:29:43.336126 3297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:43.350226 3297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:43.352221 3297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:43.352329 3297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:43.356060 3297 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:34671
I20260619 20:29:43.356117 3364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:34671 every 8 connection(s)
I20260619 20:29:43.356462 3297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:29:43.357358 3365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:43.359557 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23: Bootstrap starting.
I20260619 20:29:43.360149 3365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:43.360431 3365 log.cc:824] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:43.361125 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23: No bootstrap required, opened a new log
I20260619 20:29:43.362679 3365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3589c245539942c9aa69b1373c991a23" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34671 } }
I20260619 20:29:43.362838 3365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:43.362888 3365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3589c245539942c9aa69b1373c991a23, State: Initialized, Role: FOLLOWER
I20260619 20:29:43.363032 3365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3589c245539942c9aa69b1373c991a23" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34671 } }
I20260619 20:29:43.363113 3365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:29:43.363157 3365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:29:43.363209 3365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:43.363428 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 3297
I20260619 20:29:43.363547 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:29:43.363770 22108 external_mini_cluster.cc:1428] Setting key 81b45fdba00d872dd577d79372e2352d
I20260619 20:29:43.363940 3365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3589c245539942c9aa69b1373c991a23" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34671 } }
I20260619 20:29:43.364065 3365 leader_election.cc:302] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3589c245539942c9aa69b1373c991a23; no voters:
I20260619 20:29:43.364269 3365 leader_election.cc:288] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:29:43.364559 3365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:29:43.364884 3371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:29:43.365027 3371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [term 1 LEADER]: Becoming Leader. State: Replica: 3589c245539942c9aa69b1373c991a23, State: Running, Role: LEADER
I20260619 20:29:43.365157 3371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3589c245539942c9aa69b1373c991a23" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34671 } }
I20260619 20:29:43.365510 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3589c245539942c9aa69b1373c991a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3589c245539942c9aa69b1373c991a23" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34671 } } }
I20260619 20:29:43.365586 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:43.365712 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3589c245539942c9aa69b1373c991a23. Latest consensus state: current_term: 1 leader_uuid: "3589c245539942c9aa69b1373c991a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3589c245539942c9aa69b1373c991a23" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 34671 } } }
I20260619 20:29:43.365762 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:43.366173 3379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:29:43.366816 3379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:29:43.368397 3379 catalog_manager.cc:1347] Generated new cluster ID: 2b1507170f904f6ba2ec10ec63767ee5
I20260619 20:29:43.368449 3379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:29:43 dist-test-slave-f9rk krb5kdc[3284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:43.375218 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:43.364211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.21.151.62:34671)
0619 20:29:43.364370 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:43.364380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:43.364416 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:29:43.364538 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:43.364543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:43.364584 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:43.364733 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:43.365671 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:43.367157 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:43.368147 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:43.368741 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:43.368834 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:43.371559 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:43.371563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:43.371563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:43.371574 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:43.373435 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:43.373794 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:43.373796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:43.373797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:43.373907 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:43.374071 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:43.374072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:43.374072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:43.374334 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:43.374472 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:43.374894 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 20:29:43.375016 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260619 20:29:43.376250 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:29:43.378710 3379 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:29:43.379168 3379 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:29:43.382755 3379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3589c245539942c9aa69b1373c991a23: Generated new TSK 0
I20260619 20:29:43.382927 3379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:29:43.430299 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 3297
2026-06-19T20:29:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1343 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:29:43 dist-test-slave-f9rk krb5kdc[3392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:29:43 dist-test-slave-f9rk krb5kdc[3392](info): set up 2 sockets
Jun 19 20:29:43 dist-test-slave-f9rk krb5kdc[3392](info): commencing operation
krb5kdc: starting...
W20260619 20:29:44.464741 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:29:44 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:29:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:29:44.576241 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44171
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:44171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:44.654397 3405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:44.654631 3405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:44.654667 3405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:44.655916 3405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:29:44.655977 3405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:44.656002 3405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:44.656036 3405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:29:44.656059 3405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:29:44.657511 3405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:44171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:44.657785 3405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:44.658012 3405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:44.662879 3411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:44.662855 3410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:44.662990 3413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:44.663017 3405 server_base.cc:1034] running on GCE node
I20260619 20:29:44.663251 3405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:44.663533 3405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:44.664788 3405 hybrid_clock.cc:648] HybridClock initialized: now 1781900984664753 us; error 53 us; skew 500 ppm
Jun 19 20:29:44 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900984, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:44.667259 3405 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:29:44.667793 3405 webserver.cc:460] Webserver started at http://127.21.151.62:43533/ using document root <none> and password file <none>
I20260619 20:29:44.668042 3405 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:44.668110 3405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:44.668252 3405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:29:44.669332 3405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "a29575c2b38d40419b855d4c154fe408"
format_stamp: "Formatted at 2026-06-19 20:29:44 on dist-test-slave-f9rk"
server_key: "ba373d182469b7c2ed329463bb82a82b"
server_key_iv: "95661802fe6ca411db37db9e03c365df"
server_key_version: "encryptionkey@0"
I20260619 20:29:44.669723 3405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "a29575c2b38d40419b855d4c154fe408"
format_stamp: "Formatted at 2026-06-19 20:29:44 on dist-test-slave-f9rk"
server_key: "ba373d182469b7c2ed329463bb82a82b"
server_key_iv: "95661802fe6ca411db37db9e03c365df"
server_key_version: "encryptionkey@0"
I20260619 20:29:44.671002 3405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:29:44.671727 3420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:44.671919 3405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:29:44.672016 3405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "a29575c2b38d40419b855d4c154fe408"
format_stamp: "Formatted at 2026-06-19 20:29:44 on dist-test-slave-f9rk"
server_key: "ba373d182469b7c2ed329463bb82a82b"
server_key_iv: "95661802fe6ca411db37db9e03c365df"
server_key_version: "encryptionkey@0"
I20260619 20:29:44.672111 3405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:44.695199 3405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:44.697135 3405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:44.697299 3405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:44.701448 3405 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:44171
I20260619 20:29:44.701488 3472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:44171 every 8 connection(s)
I20260619 20:29:44.701853 3405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:29:44.702814 3473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:44.705122 3473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Bootstrap starting.
I20260619 20:29:44.705691 3473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:44.705976 3473 log.cc:824] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:44.706642 3473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: No bootstrap required, opened a new log
I20260619 20:29:44.708220 3473 raft_consensus.cc:348] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:44.708352 3473 raft_consensus.cc:374] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:44.708374 3473 raft_consensus.cc:729] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29575c2b38d40419b855d4c154fe408, State: Initialized, Role: FOLLOWER
I20260619 20:29:44.708527 3473 consensus_queue.cc:260] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:44.708586 3473 raft_consensus.cc:388] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:29:44.708606 3473 raft_consensus.cc:482] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:29:44.708666 3473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:44.709424 3473 raft_consensus.cc:504] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:44.709530 3473 leader_election.cc:302] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a29575c2b38d40419b855d4c154fe408; no voters:
I20260619 20:29:44.709693 3473 leader_election.cc:288] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:29:44.709725 3476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:29:44.709883 3476 raft_consensus.cc:686] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 1 LEADER]: Becoming Leader. State: Replica: a29575c2b38d40419b855d4c154fe408, State: Running, Role: LEADER
I20260619 20:29:44.709937 3473 sys_catalog.cc:564] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:29:44.710016 3476 consensus_queue.cc:237] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:44.710402 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a29575c2b38d40419b855d4c154fe408. Latest consensus state: current_term: 1 leader_uuid: "a29575c2b38d40419b855d4c154fe408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } } }
I20260619 20:29:44.710381 3477 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a29575c2b38d40419b855d4c154fe408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } } }
I20260619 20:29:44.710493 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:44.710566 3477 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:44.710789 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 3405
I20260619 20:29:44.710798 3485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:29:44.710893 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:29:44.711109 22108 external_mini_cluster.cc:1428] Setting key 901d17320e439de8c718be4991a88201
I20260619 20:29:44.711229 3485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:29:44.712767 3485 catalog_manager.cc:1347] Generated new cluster ID: 17dc3d6b89134eda8c61c67a6a5cb5c7
I20260619 20:29:44.712818 3485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:29:44 dist-test-slave-f9rk krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:44.723177 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:44.711665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.21.151.62:44171)
0619 20:29:44.712581 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:44.712590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:44.712624 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:29:44.712735 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:44.712743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:44.712797 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:44.712938 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:44.713620 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:44.715034 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:44.716072 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:44.716683 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:44.716830 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:44.719636 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:44.719640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:44.719640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:44.719652 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:44.721580 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:44.721977 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:44.721980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:44.721980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:44.722079 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:44.722293 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:44.722293 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:44.722294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:44.722563 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:44.722730 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:44.722838 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 20:29:44.722961 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":854,"thread_start_us":64,"threads_started":1}
I20260619 20:29:44.725375 3485 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:29:44.725833 3485 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:29:44.728588 3485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Generated new TSK 0
I20260619 20:29:44.728732 3485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:29:44.757805 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44171
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:44.854001 3499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:44.854204 3499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:44.854228 3499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:44.855393 3499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:44.855433 3499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:44.855460 3499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:29:44.856945 3499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:44.857185 3499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:44.857441 3499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:44.863549 3504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:44.863605 3499 server_base.cc:1034] running on GCE node
W20260619 20:29:44.863531 3507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:44.863814 3505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:44.864065 3499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:44.864320 3499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:44.865473 3499 hybrid_clock.cc:648] HybridClock initialized: now 1781900984865449 us; error 41 us; skew 500 ppm
Jun 19 20:29:44 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900984, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:44.868641 3499 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:29:44.869134 3499 webserver.cc:460] Webserver started at http://127.21.151.1:44115/ using document root <none> and password file <none>
I20260619 20:29:44.869391 3499 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:44.869477 3499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:44.869626 3499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:29:44.870666 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "4b5e7950e5e347c79423b7bb5a1ff71b"
format_stamp: "Formatted at 2026-06-19 20:29:44 on dist-test-slave-f9rk"
server_key: "fe728d197b1bf869d6e43b34bcde7b52"
server_key_iv: "e388ed3a49d4eb337b2e7f3c0cb69a67"
server_key_version: "encryptionkey@0"
I20260619 20:29:44.871024 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "4b5e7950e5e347c79423b7bb5a1ff71b"
format_stamp: "Formatted at 2026-06-19 20:29:44 on dist-test-slave-f9rk"
server_key: "fe728d197b1bf869d6e43b34bcde7b52"
server_key_iv: "e388ed3a49d4eb337b2e7f3c0cb69a67"
server_key_version: "encryptionkey@0"
I20260619 20:29:44.872460 3499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260619 20:29:44.873238 3514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:44.873421 3499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:29:44.873528 3499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "4b5e7950e5e347c79423b7bb5a1ff71b"
format_stamp: "Formatted at 2026-06-19 20:29:44 on dist-test-slave-f9rk"
server_key: "fe728d197b1bf869d6e43b34bcde7b52"
server_key_iv: "e388ed3a49d4eb337b2e7f3c0cb69a67"
server_key_version: "encryptionkey@0"
I20260619 20:29:44.873605 3499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:44.886333 3499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:44.888772 3499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:44.888938 3499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:44.889191 3499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:44.889537 3499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:44.889592 3499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:44.889636 3499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:44.889659 3499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:44.895548 3499 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:42831
I20260619 20:29:44.895678 3627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:42831 every 8 connection(s)
I20260619 20:29:44.896032 3499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:29:44 dist-test-slave-f9rk krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900984, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:44.903738 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 3499
I20260619 20:29:44.903834 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:29:44.904049 22108 external_mini_cluster.cc:1428] Setting key d458a7335131d243fcce111e96f45178
I20260619 20:29:44.905753 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:44.896833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:54667 (local address 127.21.151.62:44171)
0619 20:29:44.896944 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:44.896945 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:44.897144 (+ 199us) server_negotiation.cc:407] Connection header received
0619 20:29:44.897619 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:44.897620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:44.897633 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:44.897689 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:44.898657 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:44.899184 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:44.900261 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:44.900475 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:44.900561 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:44.902682 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:44.902683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:44.902684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:44.902690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:44.904267 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:44.904760 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:44.904761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:44.904761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:44.904810 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:44.905093 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:44.905094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:44.905094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:44.905211 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:44.905286 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:44.905571 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 20:29:44.905660 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 20:29:44.905990 3630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:44.896804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:54667)
0619 20:29:44.896999 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:44.897043 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:44.897537 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:44.897755 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:44.897758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:44.898348 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:44.898552 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:44.898557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:44.899266 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:44.899267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:44.900178 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:44.900180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:44.900578 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:44.900579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:44.900680 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:44.901179 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:44.901188 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:44.902595 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:44.904348 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:44.904356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:44.904358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:44.904680 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:44.904885 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:44.904886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:44.904887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:44.905016 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:44.905278 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:44.905285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:44.905435 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 20:29:44.905550 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 20:29:44.905717 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":112,"threads_started":1}
I20260619 20:29:44.906658 3628 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44171
I20260619 20:29:44.906776 3628 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:44.906992 3628 heartbeater.cc:511] Master 127.21.151.62:44171 requested a full tablet report, sending...
I20260619 20:29:44.907591 3437 ts_manager.cc:194] Registered new tserver with Master: 4b5e7950e5e347c79423b7bb5a1ff71b (127.21.151.1:42831)
I20260619 20:29:44.908334 3437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:54667
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:29:44.930009 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44171
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:45.012863 3633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:45.013073 3633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:45.013099 3633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:45.014271 3633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:45.014309 3633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:45.014335 3633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:29:45.015797 3633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:45.016000 3633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:45.016180 3633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:45.021070 3641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:45.021198 3633 server_base.cc:1034] running on GCE node
W20260619 20:29:45.021101 3639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:45.021103 3638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:45.021450 3633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:45.021708 3633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:45.022874 3633 hybrid_clock.cc:648] HybridClock initialized: now 1781900985022863 us; error 40 us; skew 500 ppm
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:45.025414 3633 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:29:45.025906 3633 webserver.cc:460] Webserver started at http://127.21.151.2:36779/ using document root <none> and password file <none>
I20260619 20:29:45.026155 3633 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:45.026223 3633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:45.026389 3633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:29:45.027441 3633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "87cca0200a3049d19710da907ce2906b"
format_stamp: "Formatted at 2026-06-19 20:29:45 on dist-test-slave-f9rk"
server_key: "a5697361628e8906a278e32c9da9c7e3"
server_key_iv: "ebc0c3410443589abd15330cb24f64eb"
server_key_version: "encryptionkey@0"
I20260619 20:29:45.027838 3633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "87cca0200a3049d19710da907ce2906b"
format_stamp: "Formatted at 2026-06-19 20:29:45 on dist-test-slave-f9rk"
server_key: "a5697361628e8906a278e32c9da9c7e3"
server_key_iv: "ebc0c3410443589abd15330cb24f64eb"
server_key_version: "encryptionkey@0"
I20260619 20:29:45.029167 3633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:29:45.029886 3648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.030077 3633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:29:45.030172 3633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "87cca0200a3049d19710da907ce2906b"
format_stamp: "Formatted at 2026-06-19 20:29:45 on dist-test-slave-f9rk"
server_key: "a5697361628e8906a278e32c9da9c7e3"
server_key_iv: "ebc0c3410443589abd15330cb24f64eb"
server_key_version: "encryptionkey@0"
I20260619 20:29:45.030256 3633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:45.043004 3633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:45.044907 3633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:45.045089 3633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:45.045322 3633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:45.045650 3633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:45.045706 3633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.045747 3633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:45.045775 3633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.051734 3633 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:38571
I20260619 20:29:45.051777 3761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:38571 every 8 connection(s)
I20260619 20:29:45.052105 3633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:29:45.054845 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 3633
I20260619 20:29:45.054956 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:29:45.055147 22108 external_mini_cluster.cc:1428] Setting key 8f43594b48a4a32c8852c906b783edc9
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:45.062275 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.052927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:38301 (local address 127.21.151.62:44171)
0619 20:29:45.053013 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.053015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.053184 (+ 169us) server_negotiation.cc:407] Connection header received
0619 20:29:45.053670 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.053671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.053685 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.053757 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:45.054676 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.055303 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.056831 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.057016 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.057073 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.059274 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:45.059275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:45.059276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:45.059284 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:45.060826 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.061336 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.061337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.061338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.061386 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.061659 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.061659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.061660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.061756 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:45.061804 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.062145 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.062211 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 20:29:45.062554 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.052885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:38301)
0619 20:29:45.053064 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.053101 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.053580 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.053828 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.053832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.054386 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:45.054596 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.054600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.055410 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.055412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.056747 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.056750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.057098 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.057099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.057253 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.057841 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:45.057850 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:45.059185 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:45.060918 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:45.060926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:45.060928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:45.061259 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:45.061453 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:45.061454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:45.061454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:45.061594 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:45.061818 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:45.061824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:45.062001 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.062129 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.062303 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260619 20:29:45.063128 3762 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44171
I20260619 20:29:45.063221 3762 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:45.063414 3762 heartbeater.cc:511] Master 127.21.151.62:44171 requested a full tablet report, sending...
I20260619 20:29:45.063805 3437 ts_manager.cc:194] Registered new tserver with Master: 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571)
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
I20260619 20:29:45.064239 3437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:38301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:29:45.081619 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44171
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:45.161680 3767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:45.161952 3767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:45.161993 3767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:45.163151 3767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:45.163223 3767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:45.163267 3767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:29:45.164724 3767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:45.165002 3767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:45.165231 3767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:45.170219 3772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:45.170254 3773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:45.170219 3775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:45.170308 3767 server_base.cc:1034] running on GCE node
I20260619 20:29:45.170671 3767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:45.170907 3767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:45.172075 3767 hybrid_clock.cc:648] HybridClock initialized: now 1781900985172066 us; error 35 us; skew 500 ppm
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:45.174479 3767 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:29:45.174975 3767 webserver.cc:460] Webserver started at http://127.21.151.3:42665/ using document root <none> and password file <none>
I20260619 20:29:45.175279 3767 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:45.175344 3767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:45.175478 3767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:29:45.176713 3767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0"
format_stamp: "Formatted at 2026-06-19 20:29:45 on dist-test-slave-f9rk"
server_key: "a9bcdc91793decde92be06c5cd452ccc"
server_key_iv: "995a9c5d080fb5d2723d8c267bfb9f90"
server_key_version: "encryptionkey@0"
I20260619 20:29:45.177086 3767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0"
format_stamp: "Formatted at 2026-06-19 20:29:45 on dist-test-slave-f9rk"
server_key: "a9bcdc91793decde92be06c5cd452ccc"
server_key_iv: "995a9c5d080fb5d2723d8c267bfb9f90"
server_key_version: "encryptionkey@0"
I20260619 20:29:45.178346 3767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:29:45.179054 3782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.179229 3767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:29:45.179323 3767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0"
format_stamp: "Formatted at 2026-06-19 20:29:45 on dist-test-slave-f9rk"
server_key: "a9bcdc91793decde92be06c5cd452ccc"
server_key_iv: "995a9c5d080fb5d2723d8c267bfb9f90"
server_key_version: "encryptionkey@0"
I20260619 20:29:45.179402 3767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:45.202702 3767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:45.204654 3767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:45.204823 3767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:45.205067 3767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:45.205390 3767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:45.205444 3767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.205518 3767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:45.205549 3767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.211414 3767 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:35475
I20260619 20:29:45.211496 3895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:35475 every 8 connection(s)
I20260619 20:29:45.211860 3767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:29:45.216529 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 3767
I20260619 20:29:45.216645 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:29:45.216815 22108 external_mini_cluster.cc:1428] Setting key 8396f6bb5317c6f4b8942cefe76f06e6
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:45.220759 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.212585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:49889 (local address 127.21.151.62:44171)
0619 20:29:45.212643 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.212646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.212857 (+ 211us) server_negotiation.cc:407] Connection header received
0619 20:29:45.213325 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.213326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.213335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.213386 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:45.214286 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.214769 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.215780 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.215958 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.216019 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.218428 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:45.218430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:45.218430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:45.218436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:45.219570 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.219967 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.219968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.219968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.220019 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.220217 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.220217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.220217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.220340 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:45.220399 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.220637 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.220697 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:29:45.220960 3898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.212560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:49889)
0619 20:29:45.212722 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.212764 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.213241 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.213456 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.213459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.214013 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:45.214208 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.214212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.214844 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.214844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.215690 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.215693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.216035 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.216036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.216130 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.216730 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:45.216739 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:45.218350 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:45.219680 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:45.219686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:45.219687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:45.219880 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:45.220074 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:45.220075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:45.220075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:45.220165 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:45.220407 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:45.220411 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:45.220528 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.220607 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.220719 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260619 20:29:45.221426 3896 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44171
I20260619 20:29:45.221499 3896 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:45.221634 3896 heartbeater.cc:511] Master 127.21.151.62:44171 requested a full tablet report, sending...
I20260619 20:29:45.221961 3437 ts_manager.cc:194] Registered new tserver with Master: 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475)
I20260619 20:29:45.222280 3437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:49889
I20260619 20:29:45.228494 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:29:45.235706 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.229949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.21.151.62:44171)
0619 20:29:45.229990 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.229992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.230088 (+ 96us) server_negotiation.cc:407] Connection header received
0619 20:29:45.230142 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.230143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.230150 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.230202 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:45.230704 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.231175 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.231938 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.232117 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.232170 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.233395 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:45.233396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:45.233397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:45.233402 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:45.234682 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.234982 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.234983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.234983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.235049 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.235206 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.235206 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.235206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.235343 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:45.235414 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.235533 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.235588 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:29:45.237325 3437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:29:45.238831 3437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:29:45.245170 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.241159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.21.151.1:42831)
0619 20:29:45.241505 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.241509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.241518 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:29:45.241555 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.241556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.241571 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.241645 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:45.241836 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.243006 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.244273 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.244760 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.244842 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.244915 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.245039 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.245079 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":73,"threads_started":1}
I20260619 20:29:45.245162 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.241054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:42831 (local address 127.0.0.1:37778)
0619 20:29:45.241237 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.241279 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.241341 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.241654 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.241657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.241659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:45.241734 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.241737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.243117 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.243118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.244186 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.244187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.244855 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.244856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.244936 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.244950 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.245023 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.245078 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
I20260619 20:29:45.245263 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.241263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:35475 (local address 127.0.0.1:57470)
0619 20:29:45.241420 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.241431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.241498 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.241761 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.241762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.241763 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:45.241804 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.241805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.243175 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.243176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.244160 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.244163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.245062 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.245063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.245125 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.245130 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.245159 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.245194 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260619 20:29:45.245256 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.241180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.21.151.2:38571)
0619 20:29:45.241365 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.241371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.241410 (+ 39us) server_negotiation.cc:407] Connection header received
0619 20:29:45.241533 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.241536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.241553 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.241608 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:45.241851 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.242643 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.244025 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.244782 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.244855 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.244910 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.245122 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.245172 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260619 20:29:45.245457 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.241065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:38571 (local address 127.0.0.1:47578)
0619 20:29:45.241348 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.241357 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.241437 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.241712 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.241713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.241715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:45.241768 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.241770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.242735 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.242736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.243957 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.243959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.244948 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.244949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.245020 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.245025 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.245353 (+ 328us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.245403 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":85,"threads_started":1}
I20260619 20:29:45.245678 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.241457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.21.151.3:35475)
0619 20:29:45.241604 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.241608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.241617 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:29:45.241654 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.241657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.241672 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.241954 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:45.242020 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.243061 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.244258 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.244973 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.245423 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.245483 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.245510 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.245623 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260619 20:29:45.246376 3830 tablet_service.cc:1467] Processing CreateTablet for tablet 2b80cf12563c4b749b2cae8f758a3989 (DEFAULT_TABLE table=test-table [id=accc4bab98cf44ac891be9f4006c0b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:29:45.246404 3696 tablet_service.cc:1467] Processing CreateTablet for tablet 2b80cf12563c4b749b2cae8f758a3989 (DEFAULT_TABLE table=test-table [id=accc4bab98cf44ac891be9f4006c0b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:29:45.246516 3562 tablet_service.cc:1467] Processing CreateTablet for tablet 2b80cf12563c4b749b2cae8f758a3989 (DEFAULT_TABLE table=test-table [id=accc4bab98cf44ac891be9f4006c0b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:29:45.246745 3696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b80cf12563c4b749b2cae8f758a3989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:45.246750 3830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b80cf12563c4b749b2cae8f758a3989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:45.246798 3562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b80cf12563c4b749b2cae8f758a3989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:45.249557 3913 tablet_bootstrap.cc:492] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: Bootstrap starting.
I20260619 20:29:45.250129 3915 tablet_bootstrap.cc:492] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Bootstrap starting.
I20260619 20:29:45.250337 3913 tablet_bootstrap.cc:654] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:45.250705 3915 tablet_bootstrap.cc:654] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:45.250892 3913 log.cc:824] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:45.251245 3915 log.cc:824] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:45.251624 3914 tablet_bootstrap.cc:492] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: Bootstrap starting.
I20260619 20:29:45.251711 3913 tablet_bootstrap.cc:492] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: No bootstrap required, opened a new log
I20260619 20:29:45.251801 3913 ts_tablet_manager.cc:1397] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:29:45.252043 3915 tablet_bootstrap.cc:492] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: No bootstrap required, opened a new log
I20260619 20:29:45.252113 3915 ts_tablet_manager.cc:1397] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:29:45.252307 3914 tablet_bootstrap.cc:654] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:45.252918 3914 log.cc:824] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:45.253726 3914 tablet_bootstrap.cc:492] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: No bootstrap required, opened a new log
I20260619 20:29:45.253795 3914 ts_tablet_manager.cc:1397] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:29:45.253705 3913 raft_consensus.cc:348] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.253916 3913 raft_consensus.cc:374] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:45.253978 3913 raft_consensus.cc:729] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7f1e2a1d4a4e449ac9ee68505041f0, State: Initialized, Role: FOLLOWER
I20260619 20:29:45.254118 3913 consensus_queue.cc:260] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.254259 3915 raft_consensus.cc:348] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.254405 3913 ts_tablet_manager.cc:1428] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 20:29:45.254436 3915 raft_consensus.cc:374] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:45.254469 3915 raft_consensus.cc:729] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e7950e5e347c79423b7bb5a1ff71b, State: Initialized, Role: FOLLOWER
I20260619 20:29:45.254556 3896 heartbeater.cc:503] Master 127.21.151.62:44171 was elected leader, sending a full tablet report...
I20260619 20:29:45.254626 3915 consensus_queue.cc:260] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.254897 3915 ts_tablet_manager.cc:1428] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:29:45.254920 3628 heartbeater.cc:503] Master 127.21.151.62:44171 was elected leader, sending a full tablet report...
I20260619 20:29:45.256172 3914 raft_consensus.cc:348] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.256304 3914 raft_consensus.cc:374] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:45.256326 3914 raft_consensus.cc:729] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87cca0200a3049d19710da907ce2906b, State: Initialized, Role: FOLLOWER
I20260619 20:29:45.256402 3914 consensus_queue.cc:260] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.256673 3914 ts_tablet_manager.cc:1428] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:29:45.256695 3762 heartbeater.cc:503] Master 127.21.151.62:44171 was elected leader, sending a full tablet report...
I20260619 20:29:45.258947 3920 raft_consensus.cc:482] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:29:45.259058 3920 raft_consensus.cc:504] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.259415 3920 leader_election.cc:288] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571), 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475)
I20260619 20:29:45.262394 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.259771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:47491 (local address 127.21.151.3:35475)
0619 20:29:45.259806 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.259808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.259974 (+ 166us) server_negotiation.cc:407] Connection header received
0619 20:29:45.260043 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.260044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.260053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.260107 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:45.260256 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.260728 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.261650 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.262028 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.262120 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.262128 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.262282 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.262316 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:29:45.262399 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.259728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:35475 (local address 127.21.151.1:47491)
0619 20:29:45.259902 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.259912 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.259983 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.260110 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.260111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.260112 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:45.260179 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.260181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.260810 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.260811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.261563 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.261565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.262121 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.262122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.262190 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.262193 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.262275 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.262316 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":2,"thread_start_us":122,"threads_started":1}
I20260619 20:29:45.262439 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.259644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:37063 (local address 127.21.151.2:38571)
0619 20:29:45.259685 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.259687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.259697 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:29:45.259812 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.259813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.259824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.259880 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:45.260053 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.260553 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.261518 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.262068 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.262166 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.262176 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.262338 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.262372 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:29:45.262485 3630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.259573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:38571 (local address 127.21.151.1:37063)
0619 20:29:45.259610 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.259643 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.259687 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.259901 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.259903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.259905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:45.259982 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.259984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.260637 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.260638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.261413 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.261415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.262166 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.262167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.262254 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.262257 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.262330 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.262372 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":22}
I20260619 20:29:45.262734 3850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b80cf12563c4b749b2cae8f758a3989" candidate_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" is_pre_election: true
I20260619 20:29:45.262841 3716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b80cf12563c4b749b2cae8f758a3989" candidate_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87cca0200a3049d19710da907ce2906b" is_pre_election: true
I20260619 20:29:45.262909 3850 raft_consensus.cc:2454] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5e7950e5e347c79423b7bb5a1ff71b in term 0.
I20260619 20:29:45.262995 3716 raft_consensus.cc:2454] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b5e7950e5e347c79423b7bb5a1ff71b in term 0.
I20260619 20:29:45.263160 3517 leader_election.cc:302] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5e7950e5e347c79423b7bb5a1ff71b, 8b7f1e2a1d4a4e449ac9ee68505041f0; no voters:
I20260619 20:29:45.263279 3920 raft_consensus.cc:2781] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:29:45.263332 3920 raft_consensus.cc:482] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:29:45.263365 3920 raft_consensus.cc:3037] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:45.264139 3920 raft_consensus.cc:504] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.264269 3920 leader_election.cc:288] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 1 election: Requested vote from peers 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571), 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475)
I20260619 20:29:45.264443 3716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b80cf12563c4b749b2cae8f758a3989" candidate_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87cca0200a3049d19710da907ce2906b"
I20260619 20:29:45.264464 3850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b80cf12563c4b749b2cae8f758a3989" candidate_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0"
I20260619 20:29:45.264524 3850 raft_consensus.cc:3037] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:45.264540 3716 raft_consensus.cc:3037] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:45.265206 3716 raft_consensus.cc:2454] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5e7950e5e347c79423b7bb5a1ff71b in term 1.
I20260619 20:29:45.265254 3850 raft_consensus.cc:2454] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b5e7950e5e347c79423b7bb5a1ff71b in term 1.
I20260619 20:29:45.265429 3517 leader_election.cc:302] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b5e7950e5e347c79423b7bb5a1ff71b, 8b7f1e2a1d4a4e449ac9ee68505041f0; no voters:
I20260619 20:29:45.265532 3920 raft_consensus.cc:2781] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:29:45.265671 3920 raft_consensus.cc:686] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 LEADER]: Becoming Leader. State: Replica: 4b5e7950e5e347c79423b7bb5a1ff71b, State: Running, Role: LEADER
I20260619 20:29:45.265766 3920 consensus_queue.cc:237] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.266532 3437 catalog_manager.cc:5495] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b5e7950e5e347c79423b7bb5a1ff71b (127.21.151.1). New cstate: current_term: 1 leader_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } health_report { overall_health: UNKNOWN } } }
I20260619 20:29:45.272228 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.270009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.21.151.1:42831)
0619 20:29:45.270072 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.270073 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.270081 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:29:45.270168 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.270169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.270202 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.270258 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:29:45.270356 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.270797 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.271498 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.271749 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.271802 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.271930 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:29:45.272021 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:29:45.272080 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.272095 (+ 15us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.272110 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:29:45.273999 3850 raft_consensus.cc:1261] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 1 FOLLOWER]: Refusing update from remote peer 4b5e7950e5e347c79423b7bb5a1ff71b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:29:45.274025 3716 raft_consensus.cc:1261] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b [term 1 FOLLOWER]: Refusing update from remote peer 4b5e7950e5e347c79423b7bb5a1ff71b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:29:45.274386 3923 consensus_queue.cc:1035] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:29:45.274572 3920 consensus_queue.cc:1035] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:29:45.277180 3926 mvcc.cc:205] Tried to move back new op lower bound from 7298666435679567872 to 7298666435649159168. Current Snapshot: MvccSnapshot[applied={T|T < 7298666435679567872}]
I20260619 20:29:45.277657 3927 mvcc.cc:205] Tried to move back new op lower bound from 7298666435679567872 to 7298666435649159168. Current Snapshot: MvccSnapshot[applied={T|T < 7298666435679567872 or (T in {7298666435679567872})}]
I20260619 20:29:45.279407 3437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:29:45.279510 3437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:29:45.280279 3437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Sending DeleteTablet for 3 replicas of tablet 2b80cf12563c4b749b2cae8f758a3989
I20260619 20:29:45.280582 3562 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b80cf12563c4b749b2cae8f758a3989 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:29:45 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:37778
I20260619 20:29:45.280665 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 3499
I20260619 20:29:45.280671 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b80cf12563c4b749b2cae8f758a3989 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:29:45 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:47578
I20260619 20:29:45.282086 3830 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b80cf12563c4b749b2cae8f758a3989 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:29:45 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57470
I20260619 20:29:45.282194 3937 tablet_replica.cc:330] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: stopping tablet replica
I20260619 20:29:45.282307 3938 tablet_replica.cc:330] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: stopping tablet replica
I20260619 20:29:45.282353 3937 raft_consensus.cc:2229] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:29:45.282424 3938 raft_consensus.cc:2229] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:29:45.282510 3937 raft_consensus.cc:2258] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:29:45.282555 3938 raft_consensus.cc:2258] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:29:45.282975 3937 ts_tablet_manager.cc:1905] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:29:45.282995 3938 ts_tablet_manager.cc:1905] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:29:45.284479 3938 ts_tablet_manager.cc:1918] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:29:45.284569 3938 log.cc:1196] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/2b80cf12563c4b749b2cae8f758a3989
I20260619 20:29:45.284847 3938 ts_tablet_manager.cc:1939] T 2b80cf12563c4b749b2cae8f758a3989 P 8b7f1e2a1d4a4e449ac9ee68505041f0: Deleting consensus metadata
I20260619 20:29:45.285044 3937 ts_tablet_manager.cc:1918] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:29:45.285122 3937 log.cc:1196] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/2b80cf12563c4b749b2cae8f758a3989
I20260619 20:29:45.285301 3423 catalog_manager.cc:4841] TS 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475): tablet 2b80cf12563c4b749b2cae8f758a3989 (table test-table [id=accc4bab98cf44ac891be9f4006c0b6d]) successfully deleted
I20260619 20:29:45.285475 3937 ts_tablet_manager.cc:1939] T 2b80cf12563c4b749b2cae8f758a3989 P 87cca0200a3049d19710da907ce2906b: Deleting consensus metadata
I20260619 20:29:45.285900 3423 catalog_manager.cc:4841] TS 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571): tablet 2b80cf12563c4b749b2cae8f758a3989 (table test-table [id=accc4bab98cf44ac891be9f4006c0b6d]) successfully deleted
W20260619 20:29:45.287099 3424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:42831 (error 108)
I20260619 20:29:45.287225 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 3633
I20260619 20:29:45.288139 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.288006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:42831 (local address 127.0.0.1:37798)
0619 20:29:45.288064 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.288102 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 20:29:45.288301 3424 catalog_manager.cc:4584] TS 4b5e7950e5e347c79423b7bb5a1ff71b (127.21.151.1:42831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b80cf12563c4b749b2cae8f758a3989: Network error: Client connection negotiation failed: client connection to 127.21.151.1:42831: connect: Connection refused (error 111)
I20260619 20:29:45.292783 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 3767
I20260619 20:29:45.297302 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 3405
I20260619 20:29:45.302290 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44171
--webserver_interface=127.21.151.62
--webserver_port=43533
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:44171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:45.383958 3940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:45.384224 3940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:45.384264 3940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:45.385506 3940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:29:45.385588 3940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:45.385612 3940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:45.385632 3940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:29:45.385663 3940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:29:45.387224 3940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:44171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=43533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:45.387554 3940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:45.387827 3940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:45.396654 3947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:45.396654 3949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:45.396657 3946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:45.396720 3940 server_base.cc:1034] running on GCE node
I20260619 20:29:45.397130 3940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:45.397431 3940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:45.398576 3940 hybrid_clock.cc:648] HybridClock initialized: now 1781900985398563 us; error 24 us; skew 500 ppm
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:45.401150 3940 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:29:45.401693 3940 webserver.cc:460] Webserver started at http://127.21.151.62:43533/ using document root <none> and password file <none>
I20260619 20:29:45.401953 3940 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:45.402011 3940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:45.403438 3940 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:29:45.404174 3956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.404368 3940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.404466 3940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "a29575c2b38d40419b855d4c154fe408"
format_stamp: "Formatted at 2026-06-19 20:29:44 on dist-test-slave-f9rk"
server_key: "ba373d182469b7c2ed329463bb82a82b"
server_key_iv: "95661802fe6ca411db37db9e03c365df"
server_key_version: "encryptionkey@0"
I20260619 20:29:45.404875 3940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:45.427974 3940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:45.430089 3940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:45.430282 3940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:45.434342 3940 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:44171
I20260619 20:29:45.434414 4008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:44171 every 8 connection(s)
I20260619 20:29:45.434829 3940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:29:45.436540 4009 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:29:45.437234 4009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Bootstrap starting.
I20260619 20:29:45.437625 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 3940
I20260619 20:29:45.437928 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:42831
--local_ip_for_outbound_sockets=127.21.151.1
--tserver_master_addrs=127.21.151.62:44171
--webserver_port=44115
--webserver_interface=127.21.151.1
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:29:45.446131 4009 log.cc:824] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:45.448117 4009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:29:45.448427 4009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Bootstrap complete.
I20260619 20:29:45.450703 4009 raft_consensus.cc:348] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:45.450958 4009 raft_consensus.cc:729] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29575c2b38d40419b855d4c154fe408, State: Initialized, Role: FOLLOWER
I20260619 20:29:45.451188 4009 consensus_queue.cc:260] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:45.451298 4009 raft_consensus.cc:388] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:29:45.451368 4009 raft_consensus.cc:482] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:29:45.451414 4009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:29:45.452497 4009 raft_consensus.cc:504] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:45.452615 4009 leader_election.cc:302] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a29575c2b38d40419b855d4c154fe408; no voters:
I20260619 20:29:45.452775 4009 leader_election.cc:288] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:29:45.452811 4013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:29:45.452912 4013 raft_consensus.cc:686] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 2 LEADER]: Becoming Leader. State: Replica: a29575c2b38d40419b855d4c154fe408, State: Running, Role: LEADER
I20260619 20:29:45.453037 4009 sys_catalog.cc:564] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:29:45.453063 4013 consensus_queue.cc:237] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:45.453701 4015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a29575c2b38d40419b855d4c154fe408. Latest consensus state: current_term: 2 leader_uuid: "a29575c2b38d40419b855d4c154fe408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } } }
I20260619 20:29:45.453785 4015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:45.453946 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a29575c2b38d40419b855d4c154fe408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } } }
I20260619 20:29:45.453994 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:45.454017 4022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:29:45.454844 4022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:29:45.454995 4022 catalog_manager.cc:1259] Loaded cluster ID: 17dc3d6b89134eda8c61c67a6a5cb5c7
I20260619 20:29:45.455245 4022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:29:45.455554 4022 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 20:29:45.460232 4022 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 20:29:45.463109 4022 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:29:45.463331 4022 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Loaded TSK: 0
I20260619 20:29:45.463562 4022 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:29:45.522917 4011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:45.523129 4011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:45.523167 4011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:45.524380 4011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:45.524451 4011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:45.524495 4011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:29:45.525949 4011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:42831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=44115
--tserver_master_addrs=127.21.151.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:45.526219 4011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:45.526448 4011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:45.532243 4037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:45.532256 4039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:45.532256 4036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:45.532495 4011 server_base.cc:1034] running on GCE node
I20260619 20:29:45.532701 4011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:45.532927 4011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:45.534070 4011 hybrid_clock.cc:648] HybridClock initialized: now 1781900985534059 us; error 31 us; skew 500 ppm
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:45.544260 4011 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:29:45.544835 4011 webserver.cc:460] Webserver started at http://127.21.151.1:44115/ using document root <none> and password file <none>
I20260619 20:29:45.545145 4011 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:45.545233 4011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:45.547004 4011 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 20:29:45.547818 4046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.548027 4011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:29:45.548122 4011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "4b5e7950e5e347c79423b7bb5a1ff71b"
format_stamp: "Formatted at 2026-06-19 20:29:44 on dist-test-slave-f9rk"
server_key: "fe728d197b1bf869d6e43b34bcde7b52"
server_key_iv: "e388ed3a49d4eb337b2e7f3c0cb69a67"
server_key_version: "encryptionkey@0"
I20260619 20:29:45.548568 4011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:45.562855 4011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:45.565271 4011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:45.565420 4011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:45.565653 4011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:45.566121 4053 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:29:45.567556 4011 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:29:45.567636 4011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:29:45.567674 4011 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:29:45.568491 4011 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:29:45.568534 4011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:29:45.568632 4053 tablet_bootstrap.cc:492] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Bootstrap starting.
I20260619 20:29:45.575033 4011 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:42831
I20260619 20:29:45.575444 4011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:29:45.578981 4053 log.cc:824] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:45.579464 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:42831 every 8 connection(s)
I20260619 20:29:45.581557 4053 tablet_bootstrap.cc:492] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:29:45.581845 4053 tablet_bootstrap.cc:492] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Bootstrap complete.
I20260619 20:29:45.582763 4053 ts_tablet_manager.cc:1397] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20260619 20:29:45.583369 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 4011
I20260619 20:29:45.583681 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:38571
--local_ip_for_outbound_sockets=127.21.151.2
--tserver_master_addrs=127.21.151.62:44171
--webserver_port=36779
--webserver_interface=127.21.151.2
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:29:45.584299 4053 raft_consensus.cc:348] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.584662 4053 raft_consensus.cc:729] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e7950e5e347c79423b7bb5a1ff71b, State: Initialized, Role: FOLLOWER
I20260619 20:29:45.584815 4053 consensus_queue.cc:260] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } }
I20260619 20:29:45.585072 4053 ts_tablet_manager.cc:1428] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:45.590969 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.576393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:56063 (local address 127.21.151.62:44171)
0619 20:29:45.579329 (+ 2936us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.579339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.579381 (+ 42us) server_negotiation.cc:407] Connection header received
0619 20:29:45.579525 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.579532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.579587 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.579749 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:45.580559 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.581878 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.583362 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.583974 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.584060 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.586916 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:45.586921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:45.586921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:45.586933 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:45.588978 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.589525 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.589528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.589528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.589638 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.589953 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.589953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.589954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.590247 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:45.590322 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.590720 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.590830 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2881,"thread_start_us":92,"threads_started":1}
I20260619 20:29:45.591069 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.576276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:56063)
0619 20:29:45.576623 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.576683 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.577404 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.579774 (+ 2370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.579778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.580151 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:45.580410 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.580417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.582019 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.582020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.583268 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.583270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.584143 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.584144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.584261 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.584985 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:45.584995 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:45.586801 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:45.589067 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:45.589074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:45.589076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:45.589430 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:45.589728 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:45.589729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:45.589730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:45.589876 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:45.590339 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:45.590345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:45.590521 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.590640 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.590826 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":96,"threads_started":1}
I20260619 20:29:45.591876 4161 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44171
I20260619 20:29:45.591969 4161 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:45.592149 4161 heartbeater.cc:511] Master 127.21.151.62:44171 requested a full tablet report, sending...
I20260619 20:29:45.592829 3973 ts_manager.cc:194] Registered new tserver with Master: 4b5e7950e5e347c79423b7bb5a1ff71b (127.21.151.1:42831)
I20260619 20:29:45.593830 3973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:56063
I20260619 20:29:45.594619 4161 heartbeater.cc:503] Master 127.21.151.62:44171 was elected leader, sending a full tablet report...
I20260619 20:29:45.599113 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.593476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.21.151.1:42831)
0619 20:29:45.595733 (+ 2257us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.595738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.595903 (+ 165us) server_negotiation.cc:407] Connection header received
0619 20:29:45.595993 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.595995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.596007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.596072 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:45.596221 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.596830 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.598306 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.598780 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.598838 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.598870 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.599039 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.599068 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2226,"thread_start_us":79,"threads_started":1}
I20260619 20:29:45.599140 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.593411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:42831 (local address 127.0.0.1:37802)
0619 20:29:45.595800 (+ 2389us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.595836 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.595910 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.596099 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.596102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.596104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:45.596160 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.596162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.596900 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.596901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.598219 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.598222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.598857 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.598858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.598936 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.598949 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.599019 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.599067 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2363,"thread_start_us":99,"threads_started":1}
I20260619 20:29:45.599529 4077 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b80cf12563c4b749b2cae8f758a3989 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:29:45 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:37802
I20260619 20:29:45.600148 4170 tablet_replica.cc:330] stopping tablet replica
I20260619 20:29:45.600256 4170 raft_consensus.cc:2229] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:29:45.600330 4170 raft_consensus.cc:2258] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:29:45.600625 4170 ts_tablet_manager.cc:1905] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:29:45.602505 4170 ts_tablet_manager.cc:1918] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:29:45.602567 4170 log.cc:1196] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/2b80cf12563c4b749b2cae8f758a3989
I20260619 20:29:45.602819 4170 ts_tablet_manager.cc:1939] T 2b80cf12563c4b749b2cae8f758a3989 P 4b5e7950e5e347c79423b7bb5a1ff71b: Deleting consensus metadata
I20260619 20:29:45.603266 3960 catalog_manager.cc:4841] TS 4b5e7950e5e347c79423b7bb5a1ff71b (127.21.151.1:42831): tablet 2b80cf12563c4b749b2cae8f758a3989 (table test-table [id=accc4bab98cf44ac891be9f4006c0b6d]) successfully deleted
W20260619 20:29:45.668594 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:45.668871 4166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:45.668908 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:45.670074 4166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:45.670132 4166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:45.670172 4166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:29:45.671527 4166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:38571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=36779
--tserver_master_addrs=127.21.151.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:45.671806 4166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:45.672025 4166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:45.676721 4178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:45.676882 4175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:45.676949 4166 server_base.cc:1034] running on GCE node
W20260619 20:29:45.676894 4176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:45.677325 4166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:45.677564 4166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:45.678725 4166 hybrid_clock.cc:648] HybridClock initialized: now 1781900985678719 us; error 38 us; skew 500 ppm
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:45.681348 4166 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:29:45.681797 4166 webserver.cc:460] Webserver started at http://127.21.151.2:36779/ using document root <none> and password file <none>
I20260619 20:29:45.682030 4166 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:45.682113 4166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:45.683455 4166 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:29:45.684114 4185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.684281 4166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.684381 4166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "87cca0200a3049d19710da907ce2906b"
format_stamp: "Formatted at 2026-06-19 20:29:45 on dist-test-slave-f9rk"
server_key: "a5697361628e8906a278e32c9da9c7e3"
server_key_iv: "ebc0c3410443589abd15330cb24f64eb"
server_key_version: "encryptionkey@0"
I20260619 20:29:45.684746 4166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:45.719455 4166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:45.721457 4166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:45.721635 4166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:45.721884 4166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:45.722200 4166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:45.722251 4166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.722292 4166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:45.722321 4166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.728130 4166 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:38571
I20260619 20:29:45.728168 4298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:38571 every 8 connection(s)
I20260619 20:29:45.728513 4166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:29:45.730374 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 4166
I20260619 20:29:45.730665 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:35475
--local_ip_for_outbound_sockets=127.21.151.3
--tserver_master_addrs=127.21.151.62:44171
--webserver_port=42665
--webserver_interface=127.21.151.3
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:45.738912 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.729238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:59521 (local address 127.21.151.62:44171)
0619 20:29:45.729304 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.729306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.729558 (+ 252us) server_negotiation.cc:407] Connection header received
0619 20:29:45.730198 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.730200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.730217 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.730297 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:45.731387 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.732172 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.733507 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.733715 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.733849 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.736103 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:45.736105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:45.736105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:45.736113 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:45.737491 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.737929 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.737930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.737931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.737991 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.738243 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.738244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.738244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.738360 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:45.738444 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.738722 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.738820 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:29:45.739117 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.729238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:59521)
0619 20:29:45.729425 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.729469 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.730087 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.730367 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.730370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.731090 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:45.731285 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.731291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.732273 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.732275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.733406 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.733409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.733863 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.733864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.734026 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.734622 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:45.734630 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:45.736001 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:45.737596 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:45.737604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:45.737606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:45.737832 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:45.738064 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:45.738065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:45.738065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:45.738174 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:45.738434 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:45.738440 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:45.738583 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.738690 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.738850 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260619 20:29:45.739754 4299 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44171
I20260619 20:29:45.739873 4299 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:45.740100 4299 heartbeater.cc:511] Master 127.21.151.62:44171 requested a full tablet report, sending...
I20260619 20:29:45.740535 3973 ts_manager.cc:194] Registered new tserver with Master: 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571)
I20260619 20:29:45.741009 3973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:59521
W20260619 20:29:45.814556 4302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:45.814769 4302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:45.814792 4302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:45.816074 4302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:45.816124 4302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:45.816174 4302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:29:45.817663 4302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:35475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=42665
--tserver_master_addrs=127.21.151.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:45.817903 4302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:45.818145 4302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:45.823019 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:45.823024 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:45.823009 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:45.823321 4302 server_base.cc:1034] running on GCE node
I20260619 20:29:45.823558 4302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:45.823825 4302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:45.824967 4302 hybrid_clock.cc:648] HybridClock initialized: now 1781900985824954 us; error 27 us; skew 500 ppm
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:45.827678 4302 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:29:45.828114 4302 webserver.cc:460] Webserver started at http://127.21.151.3:42665/ using document root <none> and password file <none>
I20260619 20:29:45.828336 4302 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:45.828392 4302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:45.829874 4302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:29:45.830513 4317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.830680 4302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:29:45.830782 4302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0"
format_stamp: "Formatted at 2026-06-19 20:29:45 on dist-test-slave-f9rk"
server_key: "a9bcdc91793decde92be06c5cd452ccc"
server_key_iv: "995a9c5d080fb5d2723d8c267bfb9f90"
server_key_version: "encryptionkey@0"
I20260619 20:29:45.831125 4302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:45.843318 4302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:45.845995 4302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:45.846199 4302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:45.846517 4302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:45.846850 4302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:45.846905 4302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.846942 4302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:45.846968 4302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.853004 4302 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:35475
I20260619 20:29:45.853045 4430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:35475 every 8 connection(s)
I20260619 20:29:45.853415 4302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:29:45.856652 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 4302
Jun 19 20:29:45 dist-test-slave-f9rk krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900985, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:29:45.863431 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.854173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:43411 (local address 127.21.151.62:44171)
0619 20:29:45.854239 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.854241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.854505 (+ 264us) server_negotiation.cc:407] Connection header received
0619 20:29:45.855064 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.855065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.855079 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.855137 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:45.856183 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.856685 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.857893 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.858126 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.858177 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.860542 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:45.860543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:45.860543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:45.860551 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:45.862003 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.862439 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.862440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.862441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.862504 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.862751 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.862752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.862753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.862867 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:45.862937 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.863259 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.863335 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:29:45.863682 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.854154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:43411)
0619 20:29:45.854380 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.854417 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.854926 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.855209 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.855220 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.855865 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:45.856067 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.856072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.856852 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.856853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.857768 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.857771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.858196 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.858197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.858294 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.858890 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:45.858899 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:45.860438 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:45.862059 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:45.862065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:45.862067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:45.862334 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:45.862571 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:45.862572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:45.862572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:45.862682 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:45.862944 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:45.862950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:45.863129 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.863238 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.863407 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":153,"threads_started":1}
I20260619 20:29:45.863785 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.857071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.21.151.62:44171)
0619 20:29:45.857206 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.857208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.857216 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:29:45.857261 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.857262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.857272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.857328 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:45.858217 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.858647 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.859424 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.859683 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.859765 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.861218 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:45.861219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:45.861219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:45.861224 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:45.862613 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.862963 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.862964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.862965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.863031 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:45.863231 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:45.863232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:45.863233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:45.863342 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:45.863516 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.863554 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.863672 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260619 20:29:45.864147 4431 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44171
I20260619 20:29:45.864269 4431 heartbeater.cc:464] Registering TS with master...
I20260619 20:29:45.864461 4431 heartbeater.cc:511] Master 127.21.151.62:44171 requested a full tablet report, sending...
I20260619 20:29:45.864840 3972 ts_manager.cc:194] Registered new tserver with Master: 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475)
I20260619 20:29:45.865204 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:43411
I20260619 20:29:45.875489 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:29:45.878377 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.875905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.21.151.62:44171)
0619 20:29:45.875983 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.875985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.876101 (+ 116us) server_negotiation.cc:407] Connection header received
0619 20:29:45.876188 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.876188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.876221 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.876293 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:29:45.876423 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.876933 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.877637 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.877822 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.877902 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.878051 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:29:45.878143 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:29:45.878251 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.878269 (+ 18us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.878291 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 20:29:45.878675 3972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:29:45.880342 3972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:29:45.882521 4077 tablet_service.cc:1467] Processing CreateTablet for tablet 40b35181203b4a179c22cb0aad85b23d (DEFAULT_TABLE table=test-table [id=587f088f288d4bb7bf5d2b8b4e800a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:29:45.882683 4077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b35181203b4a179c22cb0aad85b23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:45.884163 4053 tablet_bootstrap.cc:492] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: Bootstrap starting.
I20260619 20:29:45.884492 4053 tablet_bootstrap.cc:654] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:45.885108 4053 tablet_bootstrap.cc:492] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: No bootstrap required, opened a new log
I20260619 20:29:45.885160 4053 ts_tablet_manager.cc:1397] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 20:29:45.885257 4053 raft_consensus.cc:348] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:45.885320 4053 raft_consensus.cc:374] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:45.885357 4053 raft_consensus.cc:729] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e7950e5e347c79423b7bb5a1ff71b, State: Initialized, Role: FOLLOWER
I20260619 20:29:45.885399 4053 consensus_queue.cc:260] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:45.885492 4053 ts_tablet_manager.cc:1428] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:45.887096 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.882169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:35475 (local address 127.0.0.1:57486)
0619 20:29:45.882306 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.882317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.882356 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.883994 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.883995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.883996 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:45.884036 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.884038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.885216 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.885217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.886223 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.886226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.886839 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.886840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.886894 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.886898 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.886966 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.887033 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":12}
I20260619 20:29:45.887106 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.882291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.21.151.3:35475)
0619 20:29:45.883758 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.883764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.883775 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:29:45.883818 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.883820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.883837 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.883980 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:45.884094 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.885064 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.886299 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.886760 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.886841 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.886913 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.886983 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.887027 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1427,"thread_start_us":108,"threads_started":1}
I20260619 20:29:45.887389 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.882283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:38571 (local address 127.0.0.1:47594)
0619 20:29:45.882435 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.882442 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.882480 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.884138 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.884139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.884141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:45.884190 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.884192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.885538 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.885539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.886456 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.886458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.887136 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.887137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.887206 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.887210 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.887278 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.887319 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260619 20:29:45.887405 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.883744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.21.151.2:38571)
0619 20:29:45.883947 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.883951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.883963 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:29:45.884005 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.884007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.884023 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.884262 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:45.884360 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.885417 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.886536 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.887036 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.887121 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.887172 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.887289 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.887334 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20260619 20:29:45.887987 4365 tablet_service.cc:1467] Processing CreateTablet for tablet 40b35181203b4a179c22cb0aad85b23d (DEFAULT_TABLE table=test-table [id=587f088f288d4bb7bf5d2b8b4e800a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:29:45.888051 4233 tablet_service.cc:1467] Processing CreateTablet for tablet 40b35181203b4a179c22cb0aad85b23d (DEFAULT_TABLE table=test-table [id=587f088f288d4bb7bf5d2b8b4e800a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:29:45.888311 4365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b35181203b4a179c22cb0aad85b23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:45.888355 4233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b35181203b4a179c22cb0aad85b23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:29:45.891098 4441 tablet_bootstrap.cc:492] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: Bootstrap starting.
I20260619 20:29:45.891755 4441 tablet_bootstrap.cc:654] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:45.891767 4442 tablet_bootstrap.cc:492] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: Bootstrap starting.
I20260619 20:29:45.892226 4441 log.cc:824] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:45.892318 4442 tablet_bootstrap.cc:654] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: Neither blocks nor log segments found. Creating new log.
I20260619 20:29:45.892747 4442 log.cc:824] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:45.892968 4441 tablet_bootstrap.cc:492] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: No bootstrap required, opened a new log
I20260619 20:29:45.893045 4441 ts_tablet_manager.cc:1397] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:29:45.893450 4442 tablet_bootstrap.cc:492] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: No bootstrap required, opened a new log
I20260619 20:29:45.893528 4442 ts_tablet_manager.cc:1397] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:29:45.894680 4441 raft_consensus.cc:348] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:45.894819 4441 raft_consensus.cc:374] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:45.894842 4441 raft_consensus.cc:729] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7f1e2a1d4a4e449ac9ee68505041f0, State: Initialized, Role: FOLLOWER
I20260619 20:29:45.894982 4441 consensus_queue.cc:260] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:45.895267 4431 heartbeater.cc:503] Master 127.21.151.62:44171 was elected leader, sending a full tablet report...
I20260619 20:29:45.895268 4441 ts_tablet_manager.cc:1428] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:29:45.895263 4442 raft_consensus.cc:348] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:45.895437 4442 raft_consensus.cc:374] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:29:45.895478 4442 raft_consensus.cc:729] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87cca0200a3049d19710da907ce2906b, State: Initialized, Role: FOLLOWER
I20260619 20:29:45.895593 4442 consensus_queue.cc:260] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:45.895887 4442 ts_tablet_manager.cc:1428] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:29:45.895917 4299 heartbeater.cc:503] Master 127.21.151.62:44171 was elected leader, sending a full tablet report...
I20260619 20:29:45.935910 4446 raft_consensus.cc:482] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:29:45.936033 4446 raft_consensus.cc:504] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:45.936393 4446 leader_election.cc:288] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475), 4b5e7950e5e347c79423b7bb5a1ff71b (127.21.151.1:42831)
I20260619 20:29:45.939185 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.936622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:51269 (local address 127.21.151.3:35475)
0619 20:29:45.936688 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.936690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.936724 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:29:45.936807 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.936808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.936814 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.936863 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:45.937046 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.937537 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.938451 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.938829 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.938923 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.938932 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.939086 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.939121 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:29:45.939185 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.936590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:35475 (local address 127.21.151.2:51269)
0619 20:29:45.936662 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.936674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.936742 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.936888 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.936889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.936890 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:45.936967 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.936969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.937620 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.937621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.938358 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.938360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.938922 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.938923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.939003 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.939006 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.939068 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.939111 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 20:29:45.939493 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.936599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:42831 (local address 127.21.151.2:50521)
0619 20:29:45.936942 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.936951 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:45.937021 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:45.937168 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:45.937168 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:45.937169 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:29:45.937242 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.937243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.937863 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.937864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.938583 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:45.938585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.939150 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.939151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:45.939221 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.939224 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:29:45.939346 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 20:29:45.939398 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":91,"thread_start_us":190,"threads_started":1}
I20260619 20:29:45.939541 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.936660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:50521 (local address 127.21.151.1:42831)
0619 20:29:45.936730 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.936732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.937008 (+ 276us) server_negotiation.cc:407] Connection header received
0619 20:29:45.937098 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.937099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.937107 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.937163 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:29:45.937311 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.937769 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.938676 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.939053 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.939150 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.939158 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.939421 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.939464 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 20:29:45.939569 4385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b35181203b4a179c22cb0aad85b23d" candidate_uuid: "87cca0200a3049d19710da907ce2906b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" is_pre_election: true
I20260619 20:29:45.939786 4385 raft_consensus.cc:2454] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87cca0200a3049d19710da907ce2906b in term 0.
I20260619 20:29:45.940037 4188 leader_election.cc:302] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87cca0200a3049d19710da907ce2906b, 8b7f1e2a1d4a4e449ac9ee68505041f0; no voters:
I20260619 20:29:45.940074 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b35181203b4a179c22cb0aad85b23d" candidate_uuid: "87cca0200a3049d19710da907ce2906b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" is_pre_election: true
I20260619 20:29:45.940176 4446 raft_consensus.cc:2781] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:29:45.940228 4446 raft_consensus.cc:482] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:29:45.940241 4099 raft_consensus.cc:2454] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87cca0200a3049d19710da907ce2906b in term 0.
I20260619 20:29:45.940267 4446 raft_consensus.cc:3037] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:45.941008 4446 raft_consensus.cc:504] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:45.941164 4446 leader_election.cc:288] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [CANDIDATE]: Term 1 election: Requested vote from peers 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475), 4b5e7950e5e347c79423b7bb5a1ff71b (127.21.151.1:42831)
I20260619 20:29:45.941285 4385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b35181203b4a179c22cb0aad85b23d" candidate_uuid: "87cca0200a3049d19710da907ce2906b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0"
I20260619 20:29:45.941321 4099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b35181203b4a179c22cb0aad85b23d" candidate_uuid: "87cca0200a3049d19710da907ce2906b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b"
I20260619 20:29:45.941382 4385 raft_consensus.cc:3037] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:45.941409 4099 raft_consensus.cc:3037] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:29:45.942122 4385 raft_consensus.cc:2454] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87cca0200a3049d19710da907ce2906b in term 1.
I20260619 20:29:45.942202 4099 raft_consensus.cc:2454] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87cca0200a3049d19710da907ce2906b in term 1.
I20260619 20:29:45.942281 4188 leader_election.cc:302] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87cca0200a3049d19710da907ce2906b, 8b7f1e2a1d4a4e449ac9ee68505041f0; no voters:
I20260619 20:29:45.942375 4446 raft_consensus.cc:2781] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:29:45.942535 4446 raft_consensus.cc:686] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 1 LEADER]: Becoming Leader. State: Replica: 87cca0200a3049d19710da907ce2906b, State: Running, Role: LEADER
I20260619 20:29:45.942646 4446 consensus_queue.cc:237] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:45.943480 3973 catalog_manager.cc:5495] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b reported cstate change: term changed from 0 to 1, leader changed from <none> to 87cca0200a3049d19710da907ce2906b (127.21.151.2). New cstate: current_term: 1 leader_uuid: "87cca0200a3049d19710da907ce2906b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } health_report { overall_health: UNKNOWN } } }
I20260619 20:29:45.961007 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.958512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.21.151.2:38571)
0619 20:29:45.958578 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:45.958580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:45.958591 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:29:45.958663 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:45.958664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:45.958682 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:45.958759 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:29:45.958889 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.959361 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.960201 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:45.960403 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:45.960480 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:45.960657 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:29:45.960755 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:29:45.960846 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:45.960868 (+ 22us) server_negotiation.cc:299] Negotiation successful
0619 20:29:45.960890 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:29:45.962852 4385 raft_consensus.cc:1261] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 1 FOLLOWER]: Refusing update from remote peer 87cca0200a3049d19710da907ce2906b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:29:45.962852 4099 raft_consensus.cc:1261] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Refusing update from remote peer 87cca0200a3049d19710da907ce2906b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:29:45.963229 4448 consensus_queue.cc:1035] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:29:45.963372 4446 consensus_queue.cc:1035] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:29:45.965391 4451 mvcc.cc:205] Tried to move back new op lower bound from 7298666438501031936 to 7298666438421860352. Current Snapshot: MvccSnapshot[applied={T|T < 7298666438501031936}]
I20260619 20:29:45.965709 4453 mvcc.cc:205] Tried to move back new op lower bound from 7298666438501031936 to 7298666438421860352. Current Snapshot: MvccSnapshot[applied={T|T < 7298666438501031936}]
I20260619 20:29:45.965797 4452 mvcc.cc:205] Tried to move back new op lower bound from 7298666438501031936 to 7298666438421860352. Current Snapshot: MvccSnapshot[applied={T|T < 7298666438501031936}]
I20260619 20:29:45.968073 3973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:29:45.968168 3973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:29:45.968937 3973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Sending DeleteTablet for 3 replicas of tablet 40b35181203b4a179c22cb0aad85b23d
I20260619 20:29:45.969270 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 4011
I20260619 20:29:45.969254 4365 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b35181203b4a179c22cb0aad85b23d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:29:45 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:57486
I20260619 20:29:45.969332 4077 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b35181203b4a179c22cb0aad85b23d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:29:45 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:37802
I20260619 20:29:45.969421 4170 tablet_replica.cc:330] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: stopping tablet replica
I20260619 20:29:45.969470 4170 raft_consensus.cc:2229] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:29:45.969538 4170 raft_consensus.cc:2258] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:29:45.972091 4233 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b35181203b4a179c22cb0aad85b23d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:29:45 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:47594
I20260619 20:29:45.972347 4462 tablet_replica.cc:330] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: stopping tablet replica
I20260619 20:29:45.972468 4462 raft_consensus.cc:2229] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:29:45.972594 4462 raft_consensus.cc:2258] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:29:45.973042 4462 ts_tablet_manager.cc:1905] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:29:45.974798 4462 ts_tablet_manager.cc:1918] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:29:45.974875 4462 log.cc:1196] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/40b35181203b4a179c22cb0aad85b23d
I20260619 20:29:45.975162 4462 ts_tablet_manager.cc:1939] T 40b35181203b4a179c22cb0aad85b23d P 87cca0200a3049d19710da907ce2906b: Deleting consensus metadata
W20260619 20:29:45.975507 3960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:42831 (error 108)
I20260619 20:29:45.975571 3959 catalog_manager.cc:4841] TS 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571): tablet 40b35181203b4a179c22cb0aad85b23d (table test-table [id=587f088f288d4bb7bf5d2b8b4e800a75]) successfully deleted
I20260619 20:29:45.975883 4461 tablet_replica.cc:330] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: stopping tablet replica
I20260619 20:29:45.976013 4461 raft_consensus.cc:2229] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:29:45.976048 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:45.975934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:42831 (local address 127.0.0.1:37816)
0619 20:29:45.975981 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:45.976015 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 20:29:45.976130 4461 raft_consensus.cc:2258] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:29:45.976327 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 4166
W20260619 20:29:45.976173 3960 catalog_manager.cc:4584] TS 4b5e7950e5e347c79423b7bb5a1ff71b (127.21.151.1:42831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40b35181203b4a179c22cb0aad85b23d: Network error: Client connection negotiation failed: client connection to 127.21.151.1:42831: connect: Connection refused (error 111)
I20260619 20:29:45.977277 4461 ts_tablet_manager.cc:1905] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:29:45.979244 4461 ts_tablet_manager.cc:1918] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:29:45.979338 4461 log.cc:1196] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/40b35181203b4a179c22cb0aad85b23d
I20260619 20:29:45.979664 4461 ts_tablet_manager.cc:1939] T 40b35181203b4a179c22cb0aad85b23d P 8b7f1e2a1d4a4e449ac9ee68505041f0: Deleting consensus metadata
I20260619 20:29:45.980087 3959 catalog_manager.cc:4841] TS 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475): tablet 40b35181203b4a179c22cb0aad85b23d (table test-table [id=587f088f288d4bb7bf5d2b8b4e800a75]) successfully deleted
I20260619 20:29:45.983063 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 4302
I20260619 20:29:45.988168 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 3940
I20260619 20:29:45.993100 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44171
--webserver_interface=127.21.151.62
--webserver_port=43533
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:44171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:46.073609 4464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:46.073823 4464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:46.073845 4464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:46.074995 4464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:29:46.075035 4464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:46.075049 4464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:46.075088 4464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:29:46.075115 4464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:29:46.076581 4464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:44171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=43533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:46.076793 4464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:46.077023 4464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:46.081604 4470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:46.081590 4469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:46.081614 4472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:46.081864 4464 server_base.cc:1034] running on GCE node
I20260619 20:29:46.082093 4464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:46.082321 4464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:46.083456 4464 hybrid_clock.cc:648] HybridClock initialized: now 1781900986083448 us; error 28 us; skew 500 ppm
Jun 19 20:29:46 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900986, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:46.085916 4464 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:29:46.086366 4464 webserver.cc:460] Webserver started at http://127.21.151.62:43533/ using document root <none> and password file <none>
I20260619 20:29:46.086603 4464 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:46.086649 4464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:46.088114 4464 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:29:46.088789 4479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:46.088976 4464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:29:46.089066 4464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "a29575c2b38d40419b855d4c154fe408"
format_stamp: "Formatted at 2026-06-19 20:29:44 on dist-test-slave-f9rk"
server_key: "ba373d182469b7c2ed329463bb82a82b"
server_key_iv: "95661802fe6ca411db37db9e03c365df"
server_key_version: "encryptionkey@0"
I20260619 20:29:46.089388 4464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:46.097285 4464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:46.099452 4464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:46.099606 4464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:46.103657 4464 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:44171
I20260619 20:29:46.103690 4531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:44171 every 8 connection(s)
I20260619 20:29:46.104068 4464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:29:46.105541 4532 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:29:46.106189 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Bootstrap starting.
I20260619 20:29:46.107816 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 4464
I20260619 20:29:46.108105 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:42831
--local_ip_for_outbound_sockets=127.21.151.1
--tserver_master_addrs=127.21.151.62:44171
--webserver_port=44115
--webserver_interface=127.21.151.1
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:29:46.114454 4532 log.cc:824] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:46.116761 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:29:46.117055 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408: Bootstrap complete.
I20260619 20:29:46.119241 4532 raft_consensus.cc:348] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:46.119491 4532 raft_consensus.cc:729] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29575c2b38d40419b855d4c154fe408, State: Initialized, Role: FOLLOWER
I20260619 20:29:46.119714 4532 consensus_queue.cc:260] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:46.119796 4532 raft_consensus.cc:388] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:29:46.119848 4532 raft_consensus.cc:482] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:29:46.119897 4532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 2 FOLLOWER]: Advancing to term 3
I20260619 20:29:46.120906 4532 raft_consensus.cc:504] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:46.121028 4532 leader_election.cc:302] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a29575c2b38d40419b855d4c154fe408; no voters:
I20260619 20:29:46.121207 4532 leader_election.cc:288] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 20:29:46.121251 4536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 20:29:46.121385 4536 raft_consensus.cc:686] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [term 3 LEADER]: Becoming Leader. State: Replica: a29575c2b38d40419b855d4c154fe408, State: Running, Role: LEADER
I20260619 20:29:46.121480 4532 sys_catalog.cc:564] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:29:46.121481 4536 consensus_queue.cc:237] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } }
I20260619 20:29:46.121799 4538 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a29575c2b38d40419b855d4c154fe408. Latest consensus state: current_term: 3 leader_uuid: "a29575c2b38d40419b855d4c154fe408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } } }
I20260619 20:29:46.121863 4538 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:46.121989 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a29575c2b38d40419b855d4c154fe408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29575c2b38d40419b855d4c154fe408" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44171 } } }
I20260619 20:29:46.122053 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29575c2b38d40419b855d4c154fe408 [sys.catalog]: This master's current role is: LEADER
I20260619 20:29:46.122334 4541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:29:46.123096 4541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:29:46.123191 4541 catalog_manager.cc:1259] Loaded cluster ID: 17dc3d6b89134eda8c61c67a6a5cb5c7
I20260619 20:29:46.123252 4541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 20:29:46.123683 4541 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f84e19309ad google::LogMessage::Fail() at ??:0
@ 0x7f84e193388a google::LogMessage::SendToLog() at ??:0
@ 0x7f84e19304de google::LogMessage::Flush() at ??:0
@ 0x7f84e1931a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f84e98550f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f84e985e8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f84e37fe0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f84e37f3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f84e46576db start_thread at ??:0
@ 0x7f84e042e61f clone at ??:0
W20260619 20:29:46.206243 4534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:46.206478 4534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:46.206528 4534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:46.207747 4534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:46.207808 4534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:46.207852 4534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:29:46.209363 4534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:42831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=44115
--tserver_master_addrs=127.21.151.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:46.209642 4534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:46.209893 4534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:46.215260 4568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:46.215299 4571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:46.215348 4569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:46.215585 4534 server_base.cc:1034] running on GCE node
I20260619 20:29:46.215837 4534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:46.216150 4534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:46.217350 4534 hybrid_clock.cc:648] HybridClock initialized: now 1781900986217328 us; error 51 us; skew 500 ppm
Jun 19 20:29:46 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900986, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:46.220352 4534 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:29:46.220932 4534 webserver.cc:460] Webserver started at http://127.21.151.1:44115/ using document root <none> and password file <none>
I20260619 20:29:46.221210 4534 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:46.221272 4534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:46.223028 4534 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:29:46.223842 4578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:46.224059 4534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:29:46.224153 4534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "4b5e7950e5e347c79423b7bb5a1ff71b"
format_stamp: "Formatted at 2026-06-19 20:29:44 on dist-test-slave-f9rk"
server_key: "fe728d197b1bf869d6e43b34bcde7b52"
server_key_iv: "e388ed3a49d4eb337b2e7f3c0cb69a67"
server_key_version: "encryptionkey@0"
I20260619 20:29:46.224607 4534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:46.263403 4534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:46.265874 4534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:46.266031 4534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:46.266296 4534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:46.266781 4585 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:29:46.268301 4534 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:29:46.268362 4534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:29:46.268395 4534 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:29:46.269250 4534 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:29:46.269307 4534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:29:46.269368 4585 tablet_bootstrap.cc:492] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: Bootstrap starting.
I20260619 20:29:46.276361 4534 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:42831
I20260619 20:29:46.276602 4692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:42831 every 8 connection(s)
I20260619 20:29:46.276813 4534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:29:46.279883 4585 log.cc:824] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:29:46.281945 4585 tablet_bootstrap.cc:492] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:29:46.282230 4585 tablet_bootstrap.cc:492] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: Bootstrap complete.
I20260619 20:29:46.283138 4585 ts_tablet_manager.cc:1397] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260619 20:29:46.283515 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 4534
I20260619 20:29:46.283831 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:38571
--local_ip_for_outbound_sockets=127.21.151.2
--tserver_master_addrs=127.21.151.62:44171
--webserver_port=36779
--webserver_interface=127.21.151.2
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:29:46.284857 4585 raft_consensus.cc:348] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:46.285254 4585 raft_consensus.cc:729] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e7950e5e347c79423b7bb5a1ff71b, State: Initialized, Role: FOLLOWER
I20260619 20:29:46.285449 4585 consensus_queue.cc:260] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:46.285738 4585 ts_tablet_manager.cc:1428] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260619 20:29:46.322990 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:29:46.277589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:56715)
0619 20:29:46.277805 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:46.277853 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:46.278445 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:46.322790 (+ 44345us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
W20260619 20:29:46.323293 4693 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:44171 (0 consecutive failures): Network error: Failed to ping master at 127.21.151.62:44171: Client connection negotiation failed: client connection to 127.21.151.62:44171: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:29:46.323673 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:46.323524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:39191)
0619 20:29:46.323586 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:46.323649 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:29:46.324083 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:46.323993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:46153)
0619 20:29:46.324040 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:46.324064 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 20:29:46.324215 4693 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:29:46.366371 4697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:46.366571 4697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:46.366595 4697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:46.367833 4697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:46.367873 4697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:46.367900 4697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:29:46.369354 4697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:38571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=36779
--tserver_master_addrs=127.21.151.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:46.369614 4697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:46.369839 4697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:46.374665 4706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:46.374637 4704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:46.374907 4697 server_base.cc:1034] running on GCE node
W20260619 20:29:46.374637 4703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:46.375162 4697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:46.375392 4697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:46.376565 4697 hybrid_clock.cc:648] HybridClock initialized: now 1781900986376563 us; error 35 us; skew 500 ppm
Jun 19 20:29:46 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900986, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:46.379110 4697 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:29:46.379582 4697 webserver.cc:460] Webserver started at http://127.21.151.2:36779/ using document root <none> and password file <none>
I20260619 20:29:46.379846 4697 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:46.379914 4697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:46.381198 4697 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:29:46.381767 4713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:46.381946 4697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:29:46.382050 4697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "87cca0200a3049d19710da907ce2906b"
format_stamp: "Formatted at 2026-06-19 20:29:45 on dist-test-slave-f9rk"
server_key: "a5697361628e8906a278e32c9da9c7e3"
server_key_iv: "ebc0c3410443589abd15330cb24f64eb"
server_key_version: "encryptionkey@0"
I20260619 20:29:46.382378 4697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:46.391898 4697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:46.394186 4697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:46.394363 4697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:46.394587 4697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:46.394877 4697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:46.394930 4697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:46.394976 4697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:46.395007 4697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:46.400792 4697 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:38571
I20260619 20:29:46.400842 4826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:38571 every 8 connection(s)
I20260619 20:29:46.401170 4697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:29:46.402143 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:46.401789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:37807)
0619 20:29:46.401934 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:46.402009 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
W20260619 20:29:46.402586 4827 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:44171 (0 consecutive failures): Network error: Failed to ping master at 127.21.151.62:44171: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
I20260619 20:29:46.402858 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:46.402775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:37307)
0619 20:29:46.402817 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:46.402842 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 20:29:46.403143 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:46.403070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:45391)
0619 20:29:46.403110 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:46.403130 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 20:29:46.403264 4827 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:29:46.409338 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 4697
I20260619 20:29:46.409593 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:35475
--local_ip_for_outbound_sockets=127.21.151.3
--tserver_master_addrs=127.21.151.62:44171
--webserver_port=42665
--webserver_interface=127.21.151.3
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:29:46.489739 4830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:29:46.489974 4830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:29:46.490017 4830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:29:46.491174 4830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:29:46.491236 4830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:29:46.491282 4830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:29:46.492795 4830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:35475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=42665
--tserver_master_addrs=127.21.151.62:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:29:46.493064 4830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:29:46.493302 4830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:29:46.498193 4836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:46.498206 4838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:29:46.498220 4835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:29:46.498513 4830 server_base.cc:1034] running on GCE node
I20260619 20:29:46.498711 4830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:29:46.498953 4830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:29:46.500114 4830 hybrid_clock.cc:648] HybridClock initialized: now 1781900986500108 us; error 35 us; skew 500 ppm
Jun 19 20:29:46 dist-test-slave-f9rk krb5kdc[3392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900986, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:29:46.502645 4830 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:29:46.503093 4830 webserver.cc:460] Webserver started at http://127.21.151.3:42665/ using document root <none> and password file <none>
I20260619 20:29:46.503327 4830 fs_manager.cc:359] Metadata directory not provided
I20260619 20:29:46.503398 4830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:29:46.504834 4830 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:29:46.505386 4845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:46.505558 4830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:29:46.505656 4830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0"
format_stamp: "Formatted at 2026-06-19 20:29:45 on dist-test-slave-f9rk"
server_key: "a9bcdc91793decde92be06c5cd452ccc"
server_key_iv: "995a9c5d080fb5d2723d8c267bfb9f90"
server_key_version: "encryptionkey@0"
I20260619 20:29:46.506006 4830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:29:46.516542 4830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:29:46.518873 4830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:29:46.519042 4830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:29:46.519284 4830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:29:46.519587 4830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:29:46.519657 4830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:46.519696 4830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:29:46.519728 4830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:29:46.525532 4830 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:35475
I20260619 20:29:46.525583 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:35475 every 8 connection(s)
I20260619 20:29:46.525914 4830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:29:46.526963 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:46.526594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:38823)
0619 20:29:46.526736 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:46.526814 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
W20260619 20:29:46.527544 4959 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:44171 (0 consecutive failures): Network error: Failed to ping master at 127.21.151.62:44171: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
I20260619 20:29:46.527860 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:46.527761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:46107)
0619 20:29:46.527809 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:46.527837 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:29:46.528195 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:46.528121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:45533)
0619 20:29:46.528162 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:46.528182 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 20:29:46.528295 4959 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:29:46.534428 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 4830
I20260619 20:29:47.324798 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:47.324565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:39765)
0619 20:29:47.324678 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:47.324711 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260619 20:29:47.403815 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:47.403573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:46277)
0619 20:29:47.403734 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:47.403768 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
I20260619 20:29:47.528806 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:47.528623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:36493)
0619 20:29:47.528726 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:47.528760 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
I20260619 20:29:47.569883 4966 raft_consensus.cc:482] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:29:47.570051 4966 raft_consensus.cc:504] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:47.570374 4966 leader_election.cc:288] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475), 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571)
Jun 19 20:29:47 dist-test-slave-f9rk krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900986, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.2@KRBTEST.COM
Jun 19 20:29:47 dist-test-slave-f9rk krb5kdc[3392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781900986, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.3@KRBTEST.COM
I20260619 20:29:47.580657 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:47.570666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:38571 (local address 127.21.151.1:41443)
0619 20:29:47.570864 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:47.570874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:47.570944 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:47.571594 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:47.571595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:47.571921 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:47.572116 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:47.572122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:47.573210 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:47.573211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:47.574087 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:47.574090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:47.574716 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:47.574717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:47.574835 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:47.575495 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:47.575503 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:47.577249 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:47.579143 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:47.579150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:47.579152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:47.579385 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:47.579658 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:47.579659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:47.579659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:47.579766 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:47.580137 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:47.580143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:47.580288 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 20:29:47.580385 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 20:29:47.580552 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260619 20:29:47.580686 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:47.570740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:41443 (local address 127.21.151.2:38571)
0619 20:29:47.570848 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:47.570858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:47.570929 (+ 71us) server_negotiation.cc:407] Connection header received
0619 20:29:47.571413 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:47.571416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:47.571454 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:47.571601 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:47.572233 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:47.573106 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:47.574175 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:47.574595 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:47.574741 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:47.577361 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:47.577365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:47.577366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:47.577377 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:47.579045 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:47.579470 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:47.579474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:47.579474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:47.579563 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:47.579833 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:47.579834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:47.579835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:47.580064 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:47.580156 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:47.580458 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 20:29:47.580588 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":41,"threads_started":1}
I20260619 20:29:47.581018 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:47.570638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:40849 (local address 127.21.151.3:35475)
0619 20:29:47.570804 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 20:29:47.570818 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 20:29:47.570851 (+ 33us) server_negotiation.cc:407] Connection header received
0619 20:29:47.571278 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:29:47.571280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:29:47.571320 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:29:47.571479 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:29:47.572234 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:47.573156 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:47.574205 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:29:47.574627 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:29:47.574795 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:47.577663 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:29:47.577667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:29:47.577667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:29:47.577679 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:29:47.579388 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:47.579834 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:47.579837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:47.579838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:47.579926 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:29:47.580148 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:29:47.580148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:29:47.580149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:29:47.580506 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:29:47.580582 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:29:47.580789 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 20:29:47.580917 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
I20260619 20:29:47.581218 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:47.570537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:35475 (local address 127.21.151.1:40849)
0619 20:29:47.570587 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:47.570597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:29:47.570650 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:29:47.571504 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:29:47.571507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:29:47.571922 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:29:47.572116 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:47.572122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:47.573253 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:47.573254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:47.574089 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:29:47.574092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:29:47.574805 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:29:47.574806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:29:47.574886 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:29:47.575558 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:29:47.575563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:29:47.577564 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:29:47.579504 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:47.579505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:47.579505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:47.579744 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:47.580009 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:29:47.580009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:29:47.580010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:29:47.580070 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:29:47.580577 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:29:47.580578 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:29:47.580641 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 20:29:47.580713 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 20:29:47.581123 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":118}
W20260619 20:29:47.581694 4581 leader_election.cc:341] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571): Not found: Tablet not found: 40b35181203b4a179c22cb0aad85b23d
W20260619 20:29:47.582109 4581 leader_election.cc:341] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475): Not found: Tablet not found: 40b35181203b4a179c22cb0aad85b23d
I20260619 20:29:47.582165 4581 leader_election.cc:302] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e7950e5e347c79423b7bb5a1ff71b; no voters: 87cca0200a3049d19710da907ce2906b, 8b7f1e2a1d4a4e449ac9ee68505041f0
I20260619 20:29:47.582284 4966 raft_consensus.cc:2726] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:29:48.325402 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:48.325205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:53359)
0619 20:29:48.325320 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:48.325348 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260619 20:29:48.404431 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:48.404236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:41143)
0619 20:29:48.404353 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:48.404393 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260619 20:29:48.529474 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:48.529286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:44657)
0619 20:29:48.529397 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:48.529433 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260619 20:29:49.326395 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:49.326019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:33565)
0619 20:29:49.326299 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:49.326346 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":74,"threads_started":1}
I20260619 20:29:49.405109 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:49.404848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:33523)
0619 20:29:49.405035 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:49.405072 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":54,"threads_started":1}
I20260619 20:29:49.530263 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:49.530073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:45101)
0619 20:29:49.530188 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:49.530223 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260619 20:29:49.783514 4976 raft_consensus.cc:482] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:29:49.783659 4976 raft_consensus.cc:504] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:49.783847 4976 leader_election.cc:288] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475), 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571)
W20260619 20:29:49.784354 4581 leader_election.cc:341] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475): Not found: Tablet not found: 40b35181203b4a179c22cb0aad85b23d
W20260619 20:29:49.784418 4581 leader_election.cc:341] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571): Not found: Tablet not found: 40b35181203b4a179c22cb0aad85b23d
I20260619 20:29:49.784518 4581 leader_election.cc:302] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e7950e5e347c79423b7bb5a1ff71b; no voters: 87cca0200a3049d19710da907ce2906b, 8b7f1e2a1d4a4e449ac9ee68505041f0
I20260619 20:29:49.784627 4976 raft_consensus.cc:2726] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:29:50.327309 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:50.327087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:50201)
0619 20:29:50.327221 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:50.327262 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260619 20:29:50.405748 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:50.405555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:49283)
0619 20:29:50.405673 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:50.405709 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260619 20:29:50.530997 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:50.530758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:54819)
0619 20:29:50.530920 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:50.530956 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":59,"threads_started":1}
I20260619 20:29:51.328212 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:51.327932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:59683)
0619 20:29:51.328094 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:51.328134 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260619 20:29:51.406364 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:51.406175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:50989)
0619 20:29:51.406296 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:51.406326 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260619 20:29:51.531801 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:51.531530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:55811)
0619 20:29:51.531718 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:51.531758 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260619 20:29:52.328926 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:52.328659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:47379)
0619 20:29:52.328836 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:52.328877 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
I20260619 20:29:52.407181 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:52.406943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:55361)
0619 20:29:52.407094 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:52.407139 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260619 20:29:52.532481 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:52.532258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:59955)
0619 20:29:52.532397 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:52.532436 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260619 20:29:52.836459 4986 raft_consensus.cc:482] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:29:52.836552 4986 raft_consensus.cc:504] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:52.836747 4986 leader_election.cc:288] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475), 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571)
W20260619 20:29:52.837141 4581 leader_election.cc:341] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571): Not found: Tablet not found: 40b35181203b4a179c22cb0aad85b23d
W20260619 20:29:52.837198 4581 leader_election.cc:341] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475): Not found: Tablet not found: 40b35181203b4a179c22cb0aad85b23d
I20260619 20:29:52.837244 4581 leader_election.cc:302] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e7950e5e347c79423b7bb5a1ff71b; no voters: 87cca0200a3049d19710da907ce2906b, 8b7f1e2a1d4a4e449ac9ee68505041f0
I20260619 20:29:52.837333 4986 raft_consensus.cc:2726] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:29:53.329711 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:53.329436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:35261)
0619 20:29:53.329616 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:53.329665 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260619 20:29:53.407984 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:53.407762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:46465)
0619 20:29:53.407880 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:53.407945 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260619 20:29:53.533221 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:53.533012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:45769)
0619 20:29:53.533137 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:53.533180 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260619 20:29:54.330514 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:54.330252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:55993)
0619 20:29:54.330416 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:54.330469 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260619 20:29:54.408800 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:54.408489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:33735)
0619 20:29:54.408684 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:54.408748 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260619 20:29:54.534026 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:54.533784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:41175)
0619 20:29:54.533932 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:54.533985 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260619 20:29:54.838644 4993 raft_consensus.cc:482] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:29:54.838740 4993 raft_consensus.cc:504] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:54.838905 4993 leader_election.cc:288] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475), 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571)
W20260619 20:29:54.839365 4581 leader_election.cc:341] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475): Not found: Tablet not found: 40b35181203b4a179c22cb0aad85b23d
W20260619 20:29:54.839424 4581 leader_election.cc:341] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571): Not found: Tablet not found: 40b35181203b4a179c22cb0aad85b23d
I20260619 20:29:54.839439 4581 leader_election.cc:302] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e7950e5e347c79423b7bb5a1ff71b; no voters: 87cca0200a3049d19710da907ce2906b, 8b7f1e2a1d4a4e449ac9ee68505041f0
I20260619 20:29:54.839514 4993 raft_consensus.cc:2726] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:29:55.331254 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:55.330952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:33115)
0619 20:29:55.331168 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:55.331208 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":70,"threads_started":1}
I20260619 20:29:55.409459 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:55.409210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:44825)
0619 20:29:55.409377 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:55.409420 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260619 20:29:55.534665 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:55.534479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:54375)
0619 20:29:55.534591 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:55.534624 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260619 20:29:56.331967 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:56.331761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:44815)
0619 20:29:56.331880 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:56.331914 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260619 20:29:56.410113 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:56.409915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:52937)
0619 20:29:56.410034 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:56.410070 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260619 20:29:56.535311 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:56.535062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:46679)
0619 20:29:56.535217 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:56.535257 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260619 20:29:56.591818 5000 raft_consensus.cc:482] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:29:56.591931 5000 raft_consensus.cc:504] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f1e2a1d4a4e449ac9ee68505041f0" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 35475 } } peers { permanent_uuid: "87cca0200a3049d19710da907ce2906b" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 38571 } } peers { permanent_uuid: "4b5e7950e5e347c79423b7bb5a1ff71b" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42831 } }
I20260619 20:29:56.592116 5000 leader_election.cc:288] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475), 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571)
W20260619 20:29:56.592485 4581 leader_election.cc:341] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8b7f1e2a1d4a4e449ac9ee68505041f0 (127.21.151.3:35475): Not found: Tablet not found: 40b35181203b4a179c22cb0aad85b23d
W20260619 20:29:56.592563 4581 leader_election.cc:341] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87cca0200a3049d19710da907ce2906b (127.21.151.2:38571): Not found: Tablet not found: 40b35181203b4a179c22cb0aad85b23d
I20260619 20:29:56.592581 4581 leader_election.cc:302] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b5e7950e5e347c79423b7bb5a1ff71b; no voters: 87cca0200a3049d19710da907ce2906b, 8b7f1e2a1d4a4e449ac9ee68505041f0
I20260619 20:29:56.592638 5000 raft_consensus.cc:2726] T 40b35181203b4a179c22cb0aad85b23d P 4b5e7950e5e347c79423b7bb5a1ff71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:29:57.332620 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:57.332404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:42643)
0619 20:29:57.332539 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:57.332574 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260619 20:29:57.410799 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:57.410582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:60121)
0619 20:29:57.410717 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:57.410755 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260619 20:29:57.536072 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:57.535831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:50775)
0619 20:29:57.535986 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:57.536025 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260619 20:29:58.333305 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:58.333063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:34513)
0619 20:29:58.333212 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:58.333258 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
I20260619 20:29:58.411464 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:58.411248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:36563)
0619 20:29:58.411362 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:58.411419 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260619 20:29:58.536801 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:58.536599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:43565)
0619 20:29:58.536719 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:58.536758 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260619 20:29:59.334086 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:59.333844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:46901)
0619 20:29:59.333994 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:59.334039 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260619 20:29:59.412209 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:59.412020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:51641)
0619 20:29:59.412135 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:59.412166 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260619 20:29:59.537406 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:29:59.537179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:43207)
0619 20:29:59.537304 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:29:59.537352 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260619 20:30:00.334914 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:00.334667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:45019)
0619 20:30:00.334818 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:00.334864 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260619 20:30:00.412843 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:00.412652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:58979)
0619 20:30:00.412765 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:00.412799 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260619 20:30:00.538049 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:00.537831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:46199)
0619 20:30:00.537966 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:00.538004 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260619 20:30:01.335788 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:01.335541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.1:45553)
0619 20:30:01.335694 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:01.335738 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":112,"threads_started":1}
I20260619 20:30:01.413604 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:01.413340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.2:37935)
0619 20:30:01.413457 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:01.413530 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":82,"threads_started":1}
I20260619 20:30:01.535408 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 4534
I20260619 20:30:01.538826 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:01.538611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44171 (local address 127.21.151.3:40659)
0619 20:30:01.538736 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:01.538776 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260619 20:30:01.540284 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 4697
I20260619 20:30:01.544452 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 4830
2026-06-19T20:30:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18116 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:30:01 dist-test-slave-f9rk krb5kdc[5018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:30:01 dist-test-slave-f9rk krb5kdc[5018](info): set up 2 sockets
Jun 19 20:30:01 dist-test-slave-f9rk krb5kdc[5018](info): commencing operation
krb5kdc: starting...
W20260619 20:30:02.600896 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:30:02 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:30:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:30:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:30:02.714282 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:37727
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:37727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:30:02.794251 5031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:02.794449 5031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:02.794471 5031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:02.795668 5031 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 20:30:02.795706 5031 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 20:30:02.795719 5031 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 20:30:02.795732 5031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:30:02.795791 5031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:02.795818 5031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:02.795843 5031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:30:02.795862 5031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:30:02.797326 5031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:37727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:02.797576 5031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:02.797802 5031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:30:02.802597 5039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:02.802587 5037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:02.802647 5036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:02.802711 5031 server_base.cc:1034] running on GCE node
I20260619 20:30:02.803072 5031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:02.803328 5031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:02.804469 5031 hybrid_clock.cc:648] HybridClock initialized: now 1781901002804448 us; error 37 us; skew 500 ppm
Jun 19 20:30:02 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901002, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:02.806872 5031 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:30:02.807281 5031 webserver.cc:460] Webserver started at http://127.21.151.62:36427/ using document root <none> and password file <none>
I20260619 20:30:02.807528 5031 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:02.807597 5031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:02.807761 5031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:30:02.808959 5031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "aef803797e4a45d4a796c208ce99c58d"
format_stamp: "Formatted at 2026-06-19 20:30:02 on dist-test-slave-f9rk"
server_key: "0c4f990080a18777632ef92149c283c6"
server_key_iv: "85b1becc56a19f32261e795cd7c35856"
server_key_version: "encryptionkey@0"
I20260619 20:30:02.809311 5031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "aef803797e4a45d4a796c208ce99c58d"
format_stamp: "Formatted at 2026-06-19 20:30:02 on dist-test-slave-f9rk"
server_key: "0c4f990080a18777632ef92149c283c6"
server_key_iv: "85b1becc56a19f32261e795cd7c35856"
server_key_version: "encryptionkey@0"
I20260619 20:30:02.810616 5031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:30:02.811266 5046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:02.811409 5031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:30:02.811503 5031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "aef803797e4a45d4a796c208ce99c58d"
format_stamp: "Formatted at 2026-06-19 20:30:02 on dist-test-slave-f9rk"
server_key: "0c4f990080a18777632ef92149c283c6"
server_key_iv: "85b1becc56a19f32261e795cd7c35856"
server_key_version: "encryptionkey@0"
I20260619 20:30:02.811580 5031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:02.836696 5031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:02.838640 5031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:02.838805 5031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:02.842774 5031 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:37727
I20260619 20:30:02.842828 5098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:37727 every 8 connection(s)
I20260619 20:30:02.843199 5031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:30:02.844141 5099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:30:02.846366 5099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Bootstrap starting.
I20260619 20:30:02.846980 5099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Neither blocks nor log segments found. Creating new log.
I20260619 20:30:02.847270 5099 log.cc:824] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:30:02.848034 5099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: No bootstrap required, opened a new log
I20260619 20:30:02.849337 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 5031
I20260619 20:30:02.849458 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:30:02.849647 22108 external_mini_cluster.cc:1428] Setting key 2665b32aaa8bad5d4904d30b63e8a9ec
I20260619 20:30:02.849543 5099 raft_consensus.cc:348] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:02.849675 5099 raft_consensus.cc:374] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:30:02.849696 5099 raft_consensus.cc:729] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef803797e4a45d4a796c208ce99c58d, State: Initialized, Role: FOLLOWER
I20260619 20:30:02.849923 5099 consensus_queue.cc:260] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:02.850015 5099 raft_consensus.cc:388] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:30:02.850049 5099 raft_consensus.cc:482] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:30:02.850123 5099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:30:02.850971 5099 raft_consensus.cc:504] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:02.851056 5099 leader_election.cc:302] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aef803797e4a45d4a796c208ce99c58d; no voters:
I20260619 20:30:02.851246 5099 leader_election.cc:288] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:30:02.851276 5104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:30:02.851438 5104 raft_consensus.cc:686] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 1 LEADER]: Becoming Leader. State: Replica: aef803797e4a45d4a796c208ce99c58d, State: Running, Role: LEADER
I20260619 20:30:02.851554 5099 sys_catalog.cc:564] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:30:02.851567 5104 consensus_queue.cc:237] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:02.852052 5105 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aef803797e4a45d4a796c208ce99c58d. Latest consensus state: current_term: 1 leader_uuid: "aef803797e4a45d4a796c208ce99c58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } } }
I20260619 20:30:02.852087 5106 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aef803797e4a45d4a796c208ce99c58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } } }
I20260619 20:30:02.852187 5106 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: This master's current role is: LEADER
I20260619 20:30:02.852145 5105 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: This master's current role is: LEADER
I20260619 20:30:02.852442 5113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:30:02.853596 5113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 20:30:02 dist-test-slave-f9rk krb5kdc[5018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:30:02.857596 5113 catalog_manager.cc:1347] Generated new cluster ID: ea3cde07413c4606b1d4cca9f03724d7
I20260619 20:30:02.857653 5113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:30:02.861277 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:02.850126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.21.151.62:37727)
0619 20:30:02.850301 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:02.850311 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:02.850346 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:30:02.850457 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:02.850462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:02.850501 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:02.850630 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:02.851356 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:02.853398 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:02.854532 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:02.855107 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:02.855216 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:02.857581 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:02.857585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:02.857585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:02.857596 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:02.859574 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:02.860034 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:02.860037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:02.860037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:02.860142 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:02.860356 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:02.860357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:02.860357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:02.860664 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:02.860793 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:02.860949 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 20:30:02.861074 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260619 20:30:02.865615 5113 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:30:02.866119 5113 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:30:02.869308 5113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Generated new TSK 0
I20260619 20:30:02.869473 5113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:30:02.901697 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37727
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:30:03.003882 5125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:03.004076 5125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:03.004099 5125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:03.005252 5125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:03.005295 5125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:03.005321 5125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:30:03.006824 5125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:03.007068 5125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:03.007311 5125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:30:03.013223 5133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:03.013237 5131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:03.013237 5130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:03.013460 5125 server_base.cc:1034] running on GCE node
I20260619 20:30:03.013729 5125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:03.014036 5125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:03.015213 5125 hybrid_clock.cc:648] HybridClock initialized: now 1781901003015199 us; error 33 us; skew 500 ppm
Jun 19 20:30:03 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901003, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:03.017970 5125 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:30:03.018435 5125 webserver.cc:460] Webserver started at http://127.21.151.1:46521/ using document root <none> and password file <none>
I20260619 20:30:03.018667 5125 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:03.018733 5125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:03.018879 5125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:30:03.020017 5125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "9c11f448362e4d50be4f3b13c941cb48"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "c7b61d50aa01f26f9c6e1732d33f489b"
server_key_iv: "bb7ae5edee3fc5040591c60d774665e4"
server_key_version: "encryptionkey@0"
I20260619 20:30:03.020378 5125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "9c11f448362e4d50be4f3b13c941cb48"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "c7b61d50aa01f26f9c6e1732d33f489b"
server_key_iv: "bb7ae5edee3fc5040591c60d774665e4"
server_key_version: "encryptionkey@0"
I20260619 20:30:03.021709 5125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:03.022392 5140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:03.022544 5125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:30:03.022648 5125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "9c11f448362e4d50be4f3b13c941cb48"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "c7b61d50aa01f26f9c6e1732d33f489b"
server_key_iv: "bb7ae5edee3fc5040591c60d774665e4"
server_key_version: "encryptionkey@0"
I20260619 20:30:03.022722 5125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:03.054953 5125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:03.057468 5125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:03.057636 5125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:03.057876 5125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:30:03.058207 5125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:30:03.058269 5125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:03.058311 5125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:30:03.058339 5125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:03.064006 5125 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:37587
I20260619 20:30:03.064044 5253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:37587 every 8 connection(s)
I20260619 20:30:03.064406 5125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:30:03.067350 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 5125
I20260619 20:30:03.067464 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:30:03.067664 22108 external_mini_cluster.cc:1428] Setting key ed9c377a802bd845b6443d18f91562b1
Jun 19 20:30:03 dist-test-slave-f9rk krb5kdc[5018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901003, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:30:03.076066 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.065165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:43113 (local address 127.21.151.62:37727)
0619 20:30:03.065221 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:03.065224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:03.065391 (+ 167us) server_negotiation.cc:407] Connection header received
0619 20:30:03.066026 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:03.066028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:03.066044 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:03.066093 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:03.067020 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.067822 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.069271 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.069638 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.069715 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.072385 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:03.072387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:03.072387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:03.072398 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:03.074185 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:03.074906 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:03.074907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:03.074907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:03.074967 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:03.075363 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:03.075364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:03.075365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:03.075505 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:03.075585 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:03.075934 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 20:30:03.076001 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:30:03.076320 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.065068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:43113)
0619 20:30:03.065260 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:03.065310 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:03.065908 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:03.066163 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:03.066167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:03.066719 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:30:03.066910 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.066914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.067928 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.067929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.069176 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.069179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.069730 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.069731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.069927 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.070507 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:30:03.070519 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:30:03.072281 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:30:03.074273 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:03.074282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:03.074284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:03.074793 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:03.075054 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:03.075055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:03.075056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:03.075260 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:03.075590 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:30:03.075596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:30:03.075779 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 20:30:03.075914 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 20:30:03.076082 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
I20260619 20:30:03.076807 5254 heartbeater.cc:347] Connected to a master server at 127.21.151.62:37727
I20260619 20:30:03.076890 5254 heartbeater.cc:464] Registering TS with master...
I20260619 20:30:03.077116 5254 heartbeater.cc:511] Master 127.21.151.62:37727 requested a full tablet report, sending...
I20260619 20:30:03.077695 5063 ts_manager.cc:194] Registered new tserver with Master: 9c11f448362e4d50be4f3b13c941cb48 (127.21.151.1:37587)
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
I20260619 20:30:03.078251 5063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:43113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:30:03.094933 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37727
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:30:03.174798 5259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:03.174990 5259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:03.175012 5259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:03.176160 5259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:03.176201 5259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:03.176225 5259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:30:03.177660 5259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:03.177902 5259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:03.178154 5259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:30:03.183286 5264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:03.183295 5265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:03.183532 5259 server_base.cc:1034] running on GCE node
W20260619 20:30:03.183293 5267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:03.183936 5259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:03.184238 5259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:03.185395 5259 hybrid_clock.cc:648] HybridClock initialized: now 1781901003185384 us; error 30 us; skew 500 ppm
Jun 19 20:30:03 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901003, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:03.187783 5259 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:30:03.188246 5259 webserver.cc:460] Webserver started at http://127.21.151.2:42065/ using document root <none> and password file <none>
I20260619 20:30:03.188485 5259 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:03.188560 5259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:03.188714 5259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:30:03.189724 5259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "ed9b38a21bbf41c68e673e6acedf31d8"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "7f2bb9df2b4640cf44b791cbb96416c3"
server_key_iv: "ef43aec1b49867aab681180f5786c6f5"
server_key_version: "encryptionkey@0"
I20260619 20:30:03.190093 5259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "ed9b38a21bbf41c68e673e6acedf31d8"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "7f2bb9df2b4640cf44b791cbb96416c3"
server_key_iv: "ef43aec1b49867aab681180f5786c6f5"
server_key_version: "encryptionkey@0"
I20260619 20:30:03.191398 5259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:30:03.192166 5274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:03.192338 5259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:03.192440 5259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "ed9b38a21bbf41c68e673e6acedf31d8"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "7f2bb9df2b4640cf44b791cbb96416c3"
server_key_iv: "ef43aec1b49867aab681180f5786c6f5"
server_key_version: "encryptionkey@0"
I20260619 20:30:03.192525 5259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:03.200798 5259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:03.202651 5259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:03.202804 5259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:03.203024 5259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:30:03.203315 5259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:30:03.203368 5259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:03.203408 5259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:30:03.203436 5259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:03.209031 5259 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:45021
I20260619 20:30:03.209079 5387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:45021 every 8 connection(s)
I20260619 20:30:03.209417 5259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:30:03 dist-test-slave-f9rk krb5kdc[5018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901003, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:30:03.218529 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.210121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:37303 (local address 127.21.151.62:37727)
0619 20:30:03.210164 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:03.210166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:03.210386 (+ 220us) server_negotiation.cc:407] Connection header received
0619 20:30:03.210855 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:03.210855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:03.210865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:03.210921 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:03.211866 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.212375 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.213272 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.213457 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.213506 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.216022 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:03.216024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:03.216024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:03.216030 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:03.217174 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:03.217516 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:03.217517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:03.217517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:03.217564 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:03.217880 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:03.217881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:03.217881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:03.218002 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:03.218074 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:03.218355 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 20:30:03.218437 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:30:03.218704 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.210073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:37303)
0619 20:30:03.210259 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:03.210299 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:03.210760 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:03.210969 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:03.210973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:03.211513 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:30:03.211768 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.211773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.212449 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.212450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.213204 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.213206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.213530 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.213530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.213654 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.214165 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:30:03.214173 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:30:03.215939 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:30:03.217243 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:03.217249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:03.217250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:03.217451 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:03.217640 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:03.217641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:03.217641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:03.217774 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:03.218077 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:30:03.218083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:30:03.218213 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 20:30:03.218332 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 20:30:03.218485 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260619 20:30:03.219198 5388 heartbeater.cc:347] Connected to a master server at 127.21.151.62:37727
I20260619 20:30:03.219296 5388 heartbeater.cc:464] Registering TS with master...
I20260619 20:30:03.219504 5388 heartbeater.cc:511] Master 127.21.151.62:37727 requested a full tablet report, sending...
I20260619 20:30:03.219516 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 5259
I20260619 20:30:03.219642 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:30:03.219847 22108 external_mini_cluster.cc:1428] Setting key 550193f5016c6ae56e9dbbe1934e3ce9
I20260619 20:30:03.220041 5063 ts_manager.cc:194] Registered new tserver with Master: ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021)
I20260619 20:30:03.220554 5063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:37303
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:30:03.243588 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37727
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:30:03.323235 5393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:03.323431 5393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:03.323454 5393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:03.324669 5393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:03.324713 5393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:03.324767 5393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:30:03.326267 5393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:03.326511 5393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:03.326735 5393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:30:03.331857 5393 server_base.cc:1034] running on GCE node
W20260619 20:30:03.331810 5398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:03.331841 5399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:03.331811 5401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:03.332182 5393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:03.332460 5393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:03.333621 5393 hybrid_clock.cc:648] HybridClock initialized: now 1781901003333608 us; error 42 us; skew 500 ppm
Jun 19 20:30:03 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901003, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:03.336053 5393 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:30:03.336494 5393 webserver.cc:460] Webserver started at http://127.21.151.3:33341/ using document root <none> and password file <none>
I20260619 20:30:03.336752 5393 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:03.336823 5393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:03.336977 5393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:30:03.338011 5393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "474c4f7cd657462985769a8cde2463c5"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "d067b2f0eaef9b61cc7becf3994417ac"
server_key_iv: "5782dcb4c7a879cdca2a663b58e447db"
server_key_version: "encryptionkey@0"
I20260619 20:30:03.338374 5393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "474c4f7cd657462985769a8cde2463c5"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "d067b2f0eaef9b61cc7becf3994417ac"
server_key_iv: "5782dcb4c7a879cdca2a663b58e447db"
server_key_version: "encryptionkey@0"
I20260619 20:30:03.339715 5393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:30:03.340499 5408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:03.340677 5393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:30:03.340772 5393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "474c4f7cd657462985769a8cde2463c5"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "d067b2f0eaef9b61cc7becf3994417ac"
server_key_iv: "5782dcb4c7a879cdca2a663b58e447db"
server_key_version: "encryptionkey@0"
I20260619 20:30:03.340864 5393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:03.355288 5393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:03.357201 5393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:03.357375 5393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:03.357611 5393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:30:03.357925 5393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:30:03.357980 5393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:03.358022 5393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:30:03.358050 5393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:03.364162 5393 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:39347
I20260619 20:30:03.364224 5521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:39347 every 8 connection(s)
I20260619 20:30:03.364696 5393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:30:03.368003 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 5393
I20260619 20:30:03.368125 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:30:03.368353 22108 external_mini_cluster.cc:1428] Setting key fa4d98dac0c5b14be651c6d9b36e3d86
Jun 19 20:30:03 dist-test-slave-f9rk krb5kdc[5018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901003, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:30:03.374169 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.365404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:45633 (local address 127.21.151.62:37727)
0619 20:30:03.365447 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:03.365449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:03.365660 (+ 211us) server_negotiation.cc:407] Connection header received
0619 20:30:03.366189 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:03.366190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:03.366203 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:03.366263 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:03.367281 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.367890 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.369040 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.369314 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.369406 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.371456 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:03.371457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:03.371458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:03.371465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:03.372747 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:03.373204 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:03.373206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:03.373206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:03.373267 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:03.373514 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:03.373515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:03.373515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:03.373631 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:03.373707 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:03.373978 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 20:30:03.374059 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:30:03.374413 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.365378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:45633)
0619 20:30:03.365534 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:03.365570 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:03.366085 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:03.366336 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:03.366341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:03.366972 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:30:03.367167 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.367172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.367983 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.367984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.368948 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.368951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.369412 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.369413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.369542 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.370036 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:30:03.370044 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:30:03.371362 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:30:03.372894 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:03.372900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:03.372902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:03.373116 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:03.373334 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:03.373335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:03.373335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:03.373438 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:03.373697 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:30:03.373703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:30:03.373844 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 20:30:03.373956 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 20:30:03.374172 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260619 20:30:03.374915 5522 heartbeater.cc:347] Connected to a master server at 127.21.151.62:37727
I20260619 20:30:03.375015 5522 heartbeater.cc:464] Registering TS with master...
I20260619 20:30:03.375193 5522 heartbeater.cc:511] Master 127.21.151.62:37727 requested a full tablet report, sending...
I20260619 20:30:03.375689 5063 ts_manager.cc:194] Registered new tserver with Master: 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347)
I20260619 20:30:03.376070 5063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:45633
I20260619 20:30:03.379978 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:30:03.386704 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.381220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.21.151.62:37727)
0619 20:30:03.381294 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:03.381297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:03.381333 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:30:03.381396 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:03.381396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:03.381404 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:03.381450 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:03.381959 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.382418 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.383089 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.383264 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.383312 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.384550 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:03.384551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:03.384551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:03.384557 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:03.385721 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:03.386024 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:03.386025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:03.386025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:03.386074 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:03.386230 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:03.386231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:03.386231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:03.386373 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:03.386447 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:03.386549 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:30:03.386606 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 20:30:03.388276 5063 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:30:03.389823 5063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:30:03.396637 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.392454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.21.151.2:45021)
0619 20:30:03.392641 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:03.392646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:03.392689 (+ 43us) server_negotiation.cc:407] Connection header received
0619 20:30:03.392778 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:03.392781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:03.392797 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:03.392871 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:03.393094 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.393868 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.395443 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.396205 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.396288 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.396347 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:03.396515 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 20:30:03.396553 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260619 20:30:03.396631 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.392180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:45021 (local address 127.0.0.1:54520)
0619 20:30:03.392632 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:03.392642 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:03.392703 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:03.392966 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:03.392966 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:03.392969 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:03.393017 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.393019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.393991 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.393992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.395345 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.395347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.396290 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.396291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.396382 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.396395 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:30:03.396497 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 20:30:03.396553 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":3,"thread_start_us":56,"threads_started":1}
I20260619 20:30:03.396807 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.392359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.21.151.1:37587)
0619 20:30:03.392563 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:03.392569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:03.392581 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:30:03.392713 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:03.392716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:03.392732 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:03.392832 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:03.393031 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.394174 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.395421 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.396338 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.396403 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.396448 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:03.396684 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 20:30:03.396731 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260619 20:30:03.396795 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.392353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.21.151.3:39347)
0619 20:30:03.392679 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:03.392684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:03.393053 (+ 369us) server_negotiation.cc:407] Connection header received
0619 20:30:03.393099 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:03.393101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:03.393116 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:03.393190 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:03.393304 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.394324 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.395552 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.396283 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.396361 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.396420 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:03.396675 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 20:30:03.396717 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":80,"threads_started":1}
I20260619 20:30:03.396884 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.392186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:39347 (local address 127.0.0.1:54448)
0619 20:30:03.392938 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:03.392948 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:03.392995 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:03.393202 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:03.393203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:03.393205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:03.393243 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.393244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.394433 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.394434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.395481 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.395483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.396456 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.396457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.396521 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.396531 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 20:30:03.396583 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:30:03.396631 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":7,"thread_start_us":100,"threads_started":1}
I20260619 20:30:03.396978 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.392180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37587 (local address 127.0.0.1:54996)
0619 20:30:03.392465 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:03.392509 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:03.392576 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:03.392835 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:03.392838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:03.392840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:03.392927 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.392930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.394294 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.394295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.395325 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.395328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.396475 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.396476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.396525 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.396535 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 20:30:03.396583 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 20:30:03.396645 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":8,"thread_start_us":96,"threads_started":1}
I20260619 20:30:03.397789 5188 tablet_service.cc:1467] Processing CreateTablet for tablet 786b2e87cb114e859976c4042cdc1eb7 (DEFAULT_TABLE table=test-table [id=c1fcf6e59f774f8bb181782e9b4be839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:30:03.397812 5456 tablet_service.cc:1467] Processing CreateTablet for tablet 786b2e87cb114e859976c4042cdc1eb7 (DEFAULT_TABLE table=test-table [id=c1fcf6e59f774f8bb181782e9b4be839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:30:03.398016 5322 tablet_service.cc:1467] Processing CreateTablet for tablet 786b2e87cb114e859976c4042cdc1eb7 (DEFAULT_TABLE table=test-table [id=c1fcf6e59f774f8bb181782e9b4be839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:30:03.398182 5188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786b2e87cb114e859976c4042cdc1eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:30:03.398182 5456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786b2e87cb114e859976c4042cdc1eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:30:03.398384 5322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786b2e87cb114e859976c4042cdc1eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:30:03.401252 5539 tablet_bootstrap.cc:492] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: Bootstrap starting.
I20260619 20:30:03.401499 5541 tablet_bootstrap.cc:492] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Bootstrap starting.
I20260619 20:30:03.401944 5539 tablet_bootstrap.cc:654] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: Neither blocks nor log segments found. Creating new log.
I20260619 20:30:03.402081 5541 tablet_bootstrap.cc:654] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Neither blocks nor log segments found. Creating new log.
I20260619 20:30:03.402454 5539 log.cc:824] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:30:03.402598 5541 log.cc:824] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Log is configured to *not* fsync() on all Append() calls
I20260619 20:30:03.402941 5540 tablet_bootstrap.cc:492] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: Bootstrap starting.
I20260619 20:30:03.403198 5539 tablet_bootstrap.cc:492] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: No bootstrap required, opened a new log
I20260619 20:30:03.403270 5541 tablet_bootstrap.cc:492] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: No bootstrap required, opened a new log
I20260619 20:30:03.403357 5539 ts_tablet_manager.cc:1397] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:30:03.403405 5541 ts_tablet_manager.cc:1397] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:30:03.403677 5540 tablet_bootstrap.cc:654] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: Neither blocks nor log segments found. Creating new log.
I20260619 20:30:03.404230 5540 log.cc:824] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:30:03.405005 5540 tablet_bootstrap.cc:492] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: No bootstrap required, opened a new log
I20260619 20:30:03.405073 5540 ts_tablet_manager.cc:1397] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:30:03.405311 5541 raft_consensus.cc:348] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:03.405470 5541 raft_consensus.cc:374] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:30:03.405531 5541 raft_consensus.cc:729] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c11f448362e4d50be4f3b13c941cb48, State: Initialized, Role: FOLLOWER
I20260619 20:30:03.405524 5539 raft_consensus.cc:348] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:03.405692 5539 raft_consensus.cc:374] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:30:03.405658 5541 consensus_queue.cc:260] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:03.405779 5539 raft_consensus.cc:729] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed9b38a21bbf41c68e673e6acedf31d8, State: Initialized, Role: FOLLOWER
I20260619 20:30:03.405917 5254 heartbeater.cc:503] Master 127.21.151.62:37727 was elected leader, sending a full tablet report...
I20260619 20:30:03.405928 5541 ts_tablet_manager.cc:1428] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:30:03.405937 5539 consensus_queue.cc:260] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:03.406220 5388 heartbeater.cc:503] Master 127.21.151.62:37727 was elected leader, sending a full tablet report...
I20260619 20:30:03.406455 5539 ts_tablet_manager.cc:1428] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:30:03.407173 5540 raft_consensus.cc:348] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:03.407351 5540 raft_consensus.cc:374] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:30:03.407398 5540 raft_consensus.cc:729] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474c4f7cd657462985769a8cde2463c5, State: Initialized, Role: FOLLOWER
I20260619 20:30:03.407501 5540 consensus_queue.cc:260] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:03.407776 5540 ts_tablet_manager.cc:1428] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:30:03.407807 5522 heartbeater.cc:503] Master 127.21.151.62:37727 was elected leader, sending a full tablet report...
W20260619 20:30:03.460106 5389 tablet.cc:2369] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:30:03.565201 5255 tablet.cc:2369] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:30:03.615273 5523 tablet.cc:2369] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:30:03.755192 5546 raft_consensus.cc:482] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:30:03.755321 5546 raft_consensus.cc:504] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:03.755697 5546 leader_election.cc:288] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c11f448362e4d50be4f3b13c941cb48 (127.21.151.1:37587), 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347)
I20260619 20:30:03.759902 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.755955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:34891 (local address 127.21.151.3:39347)
0619 20:30:03.755999 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:03.756001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:03.756104 (+ 103us) server_negotiation.cc:407] Connection header received
0619 20:30:03.756158 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:03.756159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:03.756170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:03.756233 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:03.756447 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.757208 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.758930 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.759489 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.759571 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.759579 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:03.759791 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 20:30:03.759830 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:30:03.759902 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.755928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:52005 (local address 127.21.151.1:37587)
0619 20:30:03.755984 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:03.755986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:03.756105 (+ 119us) server_negotiation.cc:407] Connection header received
0619 20:30:03.756159 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:03.756160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:03.756172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:03.756233 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:03.756448 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.757205 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.758930 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.759491 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.759571 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.759579 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:03.759791 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 20:30:03.759830 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 20:30:03.759985 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.755863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37587 (local address 127.21.151.2:52005)
0619 20:30:03.756021 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:03.756031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:03.756097 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:03.756251 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:03.756252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:03.756254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:03.756365 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.756368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.757330 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.757331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.758746 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.758750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.759592 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.759593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.759702 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.759710 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 20:30:03.759770 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 20:30:03.759836 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":102,"thread_start_us":58,"threads_started":1}
I20260619 20:30:03.759987 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.755901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:39347 (local address 127.21.151.2:34891)
0619 20:30:03.756019 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:03.756030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:03.756097 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:03.756251 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:03.756252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:03.756255 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:03.756365 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.756368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.757330 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.757331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.758788 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:03.758790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.759594 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.759594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:03.759702 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.759708 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 20:30:03.759770 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:30:03.759836 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":133,"thread_start_us":58,"threads_started":1}
I20260619 20:30:03.760386 5208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786b2e87cb114e859976c4042cdc1eb7" candidate_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c11f448362e4d50be4f3b13c941cb48" is_pre_election: true
I20260619 20:30:03.760564 5208 raft_consensus.cc:2454] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed9b38a21bbf41c68e673e6acedf31d8 in term 0.
I20260619 20:30:03.760586 5476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786b2e87cb114e859976c4042cdc1eb7" candidate_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474c4f7cd657462985769a8cde2463c5" is_pre_election: true
I20260619 20:30:03.760728 5476 raft_consensus.cc:2454] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed9b38a21bbf41c68e673e6acedf31d8 in term 0.
I20260619 20:30:03.760835 5277 leader_election.cc:302] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c11f448362e4d50be4f3b13c941cb48, ed9b38a21bbf41c68e673e6acedf31d8; no voters:
I20260619 20:30:03.760984 5546 raft_consensus.cc:2781] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:30:03.761053 5546 raft_consensus.cc:482] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:30:03.761070 5546 raft_consensus.cc:3037] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:30:03.761730 5546 raft_consensus.cc:504] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:03.761879 5546 leader_election.cc:288] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [CANDIDATE]: Term 1 election: Requested vote from peers 9c11f448362e4d50be4f3b13c941cb48 (127.21.151.1:37587), 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347)
I20260619 20:30:03.762014 5208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786b2e87cb114e859976c4042cdc1eb7" candidate_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c11f448362e4d50be4f3b13c941cb48"
I20260619 20:30:03.762065 5476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786b2e87cb114e859976c4042cdc1eb7" candidate_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474c4f7cd657462985769a8cde2463c5"
I20260619 20:30:03.762097 5208 raft_consensus.cc:3037] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:30:03.762164 5476 raft_consensus.cc:3037] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:30:03.762807 5208 raft_consensus.cc:2454] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed9b38a21bbf41c68e673e6acedf31d8 in term 1.
I20260619 20:30:03.762915 5476 raft_consensus.cc:2454] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed9b38a21bbf41c68e673e6acedf31d8 in term 1.
I20260619 20:30:03.763011 5277 leader_election.cc:302] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c11f448362e4d50be4f3b13c941cb48, ed9b38a21bbf41c68e673e6acedf31d8; no voters:
I20260619 20:30:03.763096 5546 raft_consensus.cc:2781] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:30:03.763265 5546 raft_consensus.cc:686] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 1 LEADER]: Becoming Leader. State: Replica: ed9b38a21bbf41c68e673e6acedf31d8, State: Running, Role: LEADER
I20260619 20:30:03.763388 5546 consensus_queue.cc:237] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:03.764321 5062 catalog_manager.cc:5495] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2). New cstate: current_term: 1 leader_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } health_report { overall_health: HEALTHY } } }
I20260619 20:30:03.836651 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.834251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.21.151.2:45021)
0619 20:30:03.834301 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:03.834304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:03.834324 (+ 20us) server_negotiation.cc:407] Connection header received
0619 20:30:03.834369 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:03.834370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:03.834396 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:03.834472 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:30:03.834612 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.835071 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.835879 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:03.836071 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:03.836146 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:03.836297 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:30:03.836419 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:30:03.836516 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:03.836534 (+ 18us) server_negotiation.cc:299] Negotiation successful
0619 20:30:03.836553 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:30:03.838467 5476 raft_consensus.cc:1261] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5 [term 1 FOLLOWER]: Refusing update from remote peer ed9b38a21bbf41c68e673e6acedf31d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:30:03.838498 5208 raft_consensus.cc:1261] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Refusing update from remote peer ed9b38a21bbf41c68e673e6acedf31d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:30:03.838836 5550 consensus_queue.cc:1035] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:30:03.838981 5546 consensus_queue.cc:1035] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:30:03.841569 5555 mvcc.cc:205] Tried to move back new op lower bound from 7298666511719620608 to 7298666511415468032. Current Snapshot: MvccSnapshot[applied={T|T < 7298666511719620608}]
I20260619 20:30:03.842140 5554 mvcc.cc:205] Tried to move back new op lower bound from 7298666511719620608 to 7298666511415468032. Current Snapshot: MvccSnapshot[applied={T|T < 7298666511719620608}]
I20260619 20:30:03.845651 5062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:30:03.845767 5062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:30:03.846568 5062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Sending DeleteTablet for 3 replicas of tablet 786b2e87cb114e859976c4042cdc1eb7
I20260619 20:30:03.846880 5456 tablet_service.cc:1514] Processing DeleteTablet for tablet 786b2e87cb114e859976c4042cdc1eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:30:03 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:54448
I20260619 20:30:03.846915 5188 tablet_service.cc:1514] Processing DeleteTablet for tablet 786b2e87cb114e859976c4042cdc1eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:30:03 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:54996
I20260619 20:30:03.846935 5322 tablet_service.cc:1514] Processing DeleteTablet for tablet 786b2e87cb114e859976c4042cdc1eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:30:03 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:54520
I20260619 20:30:03.847118 5564 tablet_replica.cc:330] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: stopping tablet replica
I20260619 20:30:03.847177 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 5125
I20260619 20:30:03.847201 5564 raft_consensus.cc:2229] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:30:03.847313 5564 raft_consensus.cc:2258] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:30:03.848043 5564 ts_tablet_manager.cc:1905] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:30:03.850063 5564 ts_tablet_manager.cc:1918] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:30:03.850128 5564 log.cc:1196] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/786b2e87cb114e859976c4042cdc1eb7
I20260619 20:30:03.850373 5564 ts_tablet_manager.cc:1939] T 786b2e87cb114e859976c4042cdc1eb7 P ed9b38a21bbf41c68e673e6acedf31d8: Deleting consensus metadata
I20260619 20:30:03.851046 5563 tablet_replica.cc:330] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: stopping tablet replica
I20260619 20:30:03.851078 5047 catalog_manager.cc:4841] TS ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021): tablet 786b2e87cb114e859976c4042cdc1eb7 (table test-table [id=c1fcf6e59f774f8bb181782e9b4be839]) successfully deleted
I20260619 20:30:03.851199 5563 raft_consensus.cc:2229] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:30:03.851293 5563 raft_consensus.cc:2258] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 20:30:03.854120 5049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:37587 (error 108)
I20260619 20:30:03.854652 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:03.854513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37587 (local address 127.0.0.1:54998)
0619 20:30:03.854575 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:03.854613 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:37587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:30:03.854795 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 5259
W20260619 20:30:03.854822 5049 catalog_manager.cc:4584] TS 9c11f448362e4d50be4f3b13c941cb48 (127.21.151.1:37587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 786b2e87cb114e859976c4042cdc1eb7: Network error: Client connection negotiation failed: client connection to 127.21.151.1:37587: connect: Connection refused (error 111)
I20260619 20:30:03.856504 5563 ts_tablet_manager.cc:1905] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:30:03.857999 5563 ts_tablet_manager.cc:1918] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:30:03.858062 5563 log.cc:1196] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/786b2e87cb114e859976c4042cdc1eb7
I20260619 20:30:03.858302 5563 ts_tablet_manager.cc:1939] T 786b2e87cb114e859976c4042cdc1eb7 P 474c4f7cd657462985769a8cde2463c5: Deleting consensus metadata
I20260619 20:30:03.858711 5050 catalog_manager.cc:4841] TS 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347): tablet 786b2e87cb114e859976c4042cdc1eb7 (table test-table [id=c1fcf6e59f774f8bb181782e9b4be839]) successfully deleted
I20260619 20:30:03.859931 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 5393
I20260619 20:30:03.865132 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 5031
I20260619 20:30:23.870550 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:37727
--webserver_interface=127.21.151.62
--webserver_port=36427
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:37727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:30:23.952128 5567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:23.952342 5567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:23.952363 5567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:23.953444 5567 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 20:30:23.953477 5567 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 20:30:23.953490 5567 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 20:30:23.953502 5567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:30:23.953516 5567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:23.953529 5567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:23.953550 5567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:30:23.953562 5567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:30:23.955044 5567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:37727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=36427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:23.955245 5567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:23.955408 5567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:30:23.963845 5573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:23.963847 5574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:23.963869 5567 server_base.cc:1034] running on GCE node
W20260619 20:30:23.963845 5576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:23.964385 5567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:23.964737 5567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:23.965880 5567 hybrid_clock.cc:648] HybridClock initialized: now 1781901023965851 us; error 50 us; skew 500 ppm
Jun 19 20:30:23 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901023, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:23.968380 5567 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:30:23.968804 5567 webserver.cc:460] Webserver started at http://127.21.151.62:36427/ using document root <none> and password file <none>
I20260619 20:30:23.969111 5567 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:23.969193 5567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:23.970666 5567 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:23.971387 5583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:23.971599 5567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:23.971726 5567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "aef803797e4a45d4a796c208ce99c58d"
format_stamp: "Formatted at 2026-06-19 20:30:02 on dist-test-slave-f9rk"
server_key: "0c4f990080a18777632ef92149c283c6"
server_key_iv: "85b1becc56a19f32261e795cd7c35856"
server_key_version: "encryptionkey@0"
I20260619 20:30:23.972163 5567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:23.986057 5567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:23.988293 5567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:23.988464 5567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:23.992448 5567 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:37727
I20260619 20:30:23.992473 5635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:37727 every 8 connection(s)
I20260619 20:30:23.993209 5567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:30:23.994789 5636 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:30:23.995564 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 5567
I20260619 20:30:23.995570 5636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Bootstrap starting.
I20260619 20:30:23.995942 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:37587
--local_ip_for_outbound_sockets=127.21.151.1
--tserver_master_addrs=127.21.151.62:37727
--webserver_port=46521
--webserver_interface=127.21.151.1
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:30:24.004621 5636 log.cc:824] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:30:24.006562 5636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:30:24.006896 5636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Bootstrap complete.
I20260619 20:30:24.009209 5636 raft_consensus.cc:348] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:24.009526 5636 raft_consensus.cc:729] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef803797e4a45d4a796c208ce99c58d, State: Initialized, Role: FOLLOWER
I20260619 20:30:24.009699 5636 consensus_queue.cc:260] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:24.009757 5636 raft_consensus.cc:388] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:30:24.009827 5636 raft_consensus.cc:482] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:30:24.009882 5636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 1 FOLLOWER]: Advancing to term 2
I20260619 20:30:24.010934 5636 raft_consensus.cc:504] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:24.011045 5636 leader_election.cc:302] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aef803797e4a45d4a796c208ce99c58d; no voters:
I20260619 20:30:24.011209 5636 leader_election.cc:288] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 20:30:24.011261 5640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 20:30:24.011426 5640 raft_consensus.cc:686] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 2 LEADER]: Becoming Leader. State: Replica: aef803797e4a45d4a796c208ce99c58d, State: Running, Role: LEADER
I20260619 20:30:24.011493 5636 sys_catalog.cc:564] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:30:24.011564 5640 consensus_queue.cc:237] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:24.011983 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aef803797e4a45d4a796c208ce99c58d. Latest consensus state: current_term: 2 leader_uuid: "aef803797e4a45d4a796c208ce99c58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } } }
I20260619 20:30:24.012072 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: This master's current role is: LEADER
I20260619 20:30:24.011994 5641 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aef803797e4a45d4a796c208ce99c58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } } }
I20260619 20:30:24.012133 5641 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: This master's current role is: LEADER
I20260619 20:30:24.012755 5648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:30:24.013636 5648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:30:24.013733 5648 catalog_manager.cc:1259] Loaded cluster ID: ea3cde07413c4606b1d4cca9f03724d7
I20260619 20:30:24.013764 5648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:30:24.014950 5648 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:30:24.015173 5648 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Loaded TSK: 0
I20260619 20:30:24.025689 5648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Generated new TSK 1
I20260619 20:30:24.025776 5648 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:30:24.083101 5638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:24.083304 5638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:24.083328 5638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:24.084566 5638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:24.084610 5638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:24.084636 5638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:30:24.086026 5638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:37587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=46521
--tserver_master_addrs=127.21.151.62:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:24.086246 5638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:24.086488 5638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:30:24.092343 5666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:24.092569 5638 server_base.cc:1034] running on GCE node
W20260619 20:30:24.092368 5663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:24.092343 5664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:24.092823 5638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:24.093083 5638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:24.094237 5638 hybrid_clock.cc:648] HybridClock initialized: now 1781901024094231 us; error 29 us; skew 500 ppm
Jun 19 20:30:24 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901024, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:24.100617 5638 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:30:24.101197 5638 webserver.cc:460] Webserver started at http://127.21.151.1:46521/ using document root <none> and password file <none>
I20260619 20:30:24.101459 5638 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:24.101537 5638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:24.103233 5638 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 20:30:24.104008 5673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:24.104202 5638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:30:24.104306 5638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "9c11f448362e4d50be4f3b13c941cb48"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "c7b61d50aa01f26f9c6e1732d33f489b"
server_key_iv: "bb7ae5edee3fc5040591c60d774665e4"
server_key_version: "encryptionkey@0"
I20260619 20:30:24.104737 5638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:24.123855 5638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:24.126205 5638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:24.126363 5638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:24.126619 5638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:30:24.127099 5680 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:30:24.128607 5638 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:30:24.128667 5638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:30:24.128703 5638 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:30:24.129555 5638 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:30:24.129602 5638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:30:24.129665 5680 tablet_bootstrap.cc:492] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Bootstrap starting.
I20260619 20:30:24.136312 5638 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:37587
I20260619 20:30:24.136539 5787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:37587 every 8 connection(s)
I20260619 20:30:24.136739 5638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:30:24.139915 5680 log.cc:824] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Log is configured to *not* fsync() on all Append() calls
I20260619 20:30:24.141091 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 5638
I20260619 20:30:24.141402 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:45021
--local_ip_for_outbound_sockets=127.21.151.2
--tserver_master_addrs=127.21.151.62:37727
--webserver_port=42065
--webserver_interface=127.21.151.2
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:30:24.142458 5680 tablet_bootstrap.cc:492] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:30:24.142725 5680 tablet_bootstrap.cc:492] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Bootstrap complete.
I20260619 20:30:24.143913 5680 ts_tablet_manager.cc:1397] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.009s
I20260619 20:30:24.145239 5680 raft_consensus.cc:348] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:24.145560 5680 raft_consensus.cc:729] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c11f448362e4d50be4f3b13c941cb48, State: Initialized, Role: FOLLOWER
Jun 19 20:30:24 dist-test-slave-f9rk krb5kdc[5018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901024, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:30:24.146360 5680 consensus_queue.cc:260] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } }
I20260619 20:30:24.146680 5680 ts_tablet_manager.cc:1428] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 20:30:24.150640 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.137504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:59675 (local address 127.21.151.62:37727)
0619 20:30:24.138861 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.138870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.138910 (+ 40us) server_negotiation.cc:407] Connection header received
0619 20:30:24.139064 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.139073 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.139150 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.139308 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:24.140099 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.141511 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.142991 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.143604 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.144166 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.146347 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:24.146353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:24.146353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:24.146367 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:24.148657 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:24.149195 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:24.149198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:24.149198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:24.149323 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:24.149627 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:24.149628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:24.149628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:24.149897 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:24.149986 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.150334 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.150458 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304,"thread_start_us":83,"threads_started":1}
I20260619 20:30:24.150699 5790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.137432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:59675)
0619 20:30:24.137743 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.137786 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:24.138502 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:24.139356 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:24.139361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:24.139702 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:30:24.139960 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.139966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.141656 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.141657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.142885 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.142888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.143733 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.143734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.143882 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.144400 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:30:24.144409 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:30:24.146223 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:30:24.148747 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:24.148755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:24.148757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:24.149106 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:24.149395 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:24.149396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:24.149397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:24.149552 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:24.149983 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:30:24.149990 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:30:24.150154 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 20:30:24.150278 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 20:30:24.150467 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":104,"threads_started":1}
I20260619 20:30:24.151749 5788 heartbeater.cc:347] Connected to a master server at 127.21.151.62:37727
I20260619 20:30:24.151851 5788 heartbeater.cc:464] Registering TS with master...
I20260619 20:30:24.152037 5788 heartbeater.cc:511] Master 127.21.151.62:37727 requested a full tablet report, sending...
I20260619 20:30:24.152920 5600 ts_manager.cc:194] Registered new tserver with Master: 9c11f448362e4d50be4f3b13c941cb48 (127.21.151.1:37587)
I20260619 20:30:24.154043 5600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:59675
I20260619 20:30:24.155002 5788 heartbeater.cc:503] Master 127.21.151.62:37727 was elected leader, sending a full tablet report...
I20260619 20:30:24.159243 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.153575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37587 (local address 127.0.0.1:49458)
0619 20:30:24.155720 (+ 2145us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.155757 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:24.155839 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:24.156041 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:24.156044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:24.156046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:24.156111 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.156113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.157335 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.157337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.158128 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.158130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.159006 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.159007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.159072 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.159081 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 20:30:24.159148 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 20:30:24.159187 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2114,"thread_start_us":84,"threads_started":1}
I20260619 20:30:24.159277 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.153815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.21.151.1:37587)
0619 20:30:24.153956 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.153960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.155834 (+ 1874us) server_negotiation.cc:407] Connection header received
0619 20:30:24.155923 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.155925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.155941 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.156019 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:24.156185 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.157250 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.158205 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.158911 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.158984 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.159036 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.159173 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.159211 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
I20260619 20:30:24.159809 5722 tablet_service.cc:1514] Processing DeleteTablet for tablet 786b2e87cb114e859976c4042cdc1eb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:30:03 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:49458
I20260619 20:30:24.160359 5797 tablet_replica.cc:330] stopping tablet replica
I20260619 20:30:24.160485 5797 raft_consensus.cc:2229] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:30:24.160552 5797 raft_consensus.cc:2258] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:30:24.160836 5797 ts_tablet_manager.cc:1905] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:30:24.162755 5797 ts_tablet_manager.cc:1918] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:30:24.162820 5797 log.cc:1196] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal/wals/786b2e87cb114e859976c4042cdc1eb7
I20260619 20:30:24.163055 5797 ts_tablet_manager.cc:1939] T 786b2e87cb114e859976c4042cdc1eb7 P 9c11f448362e4d50be4f3b13c941cb48: Deleting consensus metadata
I20260619 20:30:24.163558 5586 catalog_manager.cc:4841] TS 9c11f448362e4d50be4f3b13c941cb48 (127.21.151.1:37587): tablet 786b2e87cb114e859976c4042cdc1eb7 (table test-table [id=c1fcf6e59f774f8bb181782e9b4be839]) successfully deleted
W20260619 20:30:24.227248 5793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:24.227481 5793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:24.227517 5793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:24.228668 5793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:24.228747 5793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:24.228787 5793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:30:24.230121 5793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:45021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=42065
--tserver_master_addrs=127.21.151.62:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:24.230366 5793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:24.230574 5793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:30:24.235530 5793 server_base.cc:1034] running on GCE node
W20260619 20:30:24.235492 5803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:24.235505 5802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:24.235502 5805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:24.235988 5793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:24.236250 5793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:24.237411 5793 hybrid_clock.cc:648] HybridClock initialized: now 1781901024237399 us; error 30 us; skew 500 ppm
Jun 19 20:30:24 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901024, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:24.239773 5793 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:30:24.240221 5793 webserver.cc:460] Webserver started at http://127.21.151.2:42065/ using document root <none> and password file <none>
I20260619 20:30:24.240485 5793 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:24.240576 5793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:24.241849 5793 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:24.242543 5812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:24.242751 5793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:24.242846 5793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "ed9b38a21bbf41c68e673e6acedf31d8"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "7f2bb9df2b4640cf44b791cbb96416c3"
server_key_iv: "ef43aec1b49867aab681180f5786c6f5"
server_key_version: "encryptionkey@0"
I20260619 20:30:24.243170 5793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:24.254887 5793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:24.256812 5793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:24.256986 5793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:24.257230 5793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:30:24.257532 5793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:30:24.257586 5793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:24.257625 5793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:30:24.257653 5793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:24.263643 5793 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:45021
I20260619 20:30:24.263685 5925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:45021 every 8 connection(s)
I20260619 20:30:24.264014 5793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:30:24.268922 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 5793
I20260619 20:30:24.269254 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:39347
--local_ip_for_outbound_sockets=127.21.151.3
--tserver_master_addrs=127.21.151.62:37727
--webserver_port=33341
--webserver_interface=127.21.151.3
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 20:30:24 dist-test-slave-f9rk krb5kdc[5018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901024, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:30:24.273886 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.264709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:51251 (local address 127.21.151.62:37727)
0619 20:30:24.264757 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.264759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.264966 (+ 207us) server_negotiation.cc:407] Connection header received
0619 20:30:24.265444 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.265445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.265456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.265506 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:24.266389 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.266877 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.268017 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.268208 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.268261 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.270806 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:24.270807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:24.270808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:24.270816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:24.272389 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:24.272877 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:24.272877 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:24.272878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:24.272932 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:24.273254 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:24.273255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:24.273255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:24.273356 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:24.273404 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.273754 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.273823 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:30:24.274235 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.264693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:51251)
0619 20:30:24.264839 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.264874 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:24.265355 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:24.265562 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:24.265565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:24.266105 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:30:24.266304 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.266309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.266975 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.266976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.267939 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.267942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.268280 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.268281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.268380 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.268847 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:30:24.268858 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:30:24.270704 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:30:24.272482 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:24.272489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:24.272491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:24.272798 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:24.273000 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:24.273001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:24.273001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:24.273151 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:24.273422 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:30:24.273429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:30:24.273608 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 20:30:24.273739 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 20:30:24.273928 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260619 20:30:24.274791 5926 heartbeater.cc:347] Connected to a master server at 127.21.151.62:37727
I20260619 20:30:24.274894 5926 heartbeater.cc:464] Registering TS with master...
I20260619 20:30:24.275126 5926 heartbeater.cc:511] Master 127.21.151.62:37727 requested a full tablet report, sending...
I20260619 20:30:24.275511 5600 ts_manager.cc:194] Registered new tserver with Master: ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021)
I20260619 20:30:24.275900 5600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:51251
W20260619 20:30:24.351380 5929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:24.351575 5929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:24.351598 5929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:24.352774 5929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:24.352818 5929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:24.352872 5929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:30:24.354292 5929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:39347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=33341
--tserver_master_addrs=127.21.151.62:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:24.354522 5929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:24.354758 5929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:30:24.359561 5934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:24.359570 5935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:24.359790 5929 server_base.cc:1034] running on GCE node
W20260619 20:30:24.359570 5937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:24.360044 5929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:24.360313 5929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:24.361474 5929 hybrid_clock.cc:648] HybridClock initialized: now 1781901024361473 us; error 38 us; skew 500 ppm
Jun 19 20:30:24 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901024, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:24.363832 5929 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:30:24.364270 5929 webserver.cc:460] Webserver started at http://127.21.151.3:33341/ using document root <none> and password file <none>
I20260619 20:30:24.364554 5929 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:24.364639 5929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:24.365953 5929 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:30:24.366523 5944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:24.366689 5929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:30:24.366783 5929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "474c4f7cd657462985769a8cde2463c5"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "d067b2f0eaef9b61cc7becf3994417ac"
server_key_iv: "5782dcb4c7a879cdca2a663b58e447db"
server_key_version: "encryptionkey@0"
I20260619 20:30:24.367131 5929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:24.374475 5929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:24.376430 5929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:24.376579 5929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:24.376780 5929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:30:24.377074 5929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:30:24.377131 5929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:24.377169 5929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:30:24.377200 5929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:24.383412 5929 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:39347
I20260619 20:30:24.383463 6057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:39347 every 8 connection(s)
I20260619 20:30:24.383786 5929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:30:24.384833 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 5929
Jun 19 20:30:24 dist-test-slave-f9rk krb5kdc[5018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901024, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:30:24.392408 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.385313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.21.151.62:37727)
0619 20:30:24.385614 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.385616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.385622 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:30:24.385642 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.385643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.385654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.385725 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:24.386302 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.386947 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.387845 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.388150 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.388223 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.389641 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:24.389642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:24.389642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:24.389649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:24.391175 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:24.391577 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:24.391578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:24.391578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:24.391676 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:24.391940 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:24.391941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:24.391941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:24.392043 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:24.392158 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.392256 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.392340 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":51,"threads_started":1}
I20260619 20:30:24.393729 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.384463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:33611 (local address 127.21.151.62:37727)
0619 20:30:24.384550 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.384553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.384745 (+ 192us) server_negotiation.cc:407] Connection header received
0619 20:30:24.385488 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.385490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.385499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.385535 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:24.386664 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.387206 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.388494 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.388693 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.388769 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.390955 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:24.390956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:24.390956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:24.390963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:24.392588 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:24.392962 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:24.392963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:24.392963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:24.393019 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:24.393272 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:24.393272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:24.393273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:24.393363 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:24.393415 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.393622 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.393675 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:30:24.393986 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.384424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:33611)
0619 20:30:24.384594 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.384642 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:24.385384 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:24.385626 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:24.385631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:24.386316 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:30:24.386572 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.386579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.387297 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.387298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.388407 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.388409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.388781 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.388782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.388922 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.389366 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:30:24.389374 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:30:24.390851 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:30:24.392668 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:24.392675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:24.392677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:24.392911 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:24.393095 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:24.393095 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:24.393096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:24.393222 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:24.393421 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:30:24.393426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:30:24.393533 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 20:30:24.393612 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:30:24.393723 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260619 20:30:24.394479 6058 heartbeater.cc:347] Connected to a master server at 127.21.151.62:37727
I20260619 20:30:24.394583 6058 heartbeater.cc:464] Registering TS with master...
I20260619 20:30:24.394796 6058 heartbeater.cc:511] Master 127.21.151.62:37727 requested a full tablet report, sending...
I20260619 20:30:24.395169 5600 ts_manager.cc:194] Registered new tserver with Master: 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347)
I20260619 20:30:24.395519 5600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:33611
I20260619 20:30:24.404198 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:30:24.407078 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.404533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.21.151.62:37727)
0619 20:30:24.404605 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.404607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.404704 (+ 97us) server_negotiation.cc:407] Connection header received
0619 20:30:24.404800 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.404801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.404847 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.404922 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:30:24.405060 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.405562 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.406343 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.406539 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.406614 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.406773 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:30:24.406823 (+ 50us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 20:30:24.407003 (+ 180us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57844: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 20:30:24.407126 5791 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57844: token expired
W20260619 20:30:24.407202 5527 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.151.62:37727)
I20260619 20:30:24.413123 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.407447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.21.151.62:37727)
0619 20:30:24.407478 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.407480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.407511 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:30:24.407569 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.407570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.407576 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.407662 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:24.408224 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.408671 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.409384 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.409560 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.409614 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.410684 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:24.410685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:24.410685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:24.410691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:24.412187 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:24.412458 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:24.412458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:24.412459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:24.412506 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:24.412698 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:24.412698 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:24.412699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:24.412832 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:24.412907 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.413014 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.413066 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 20:30:24.414191 5600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:30:24.415779 5600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:30:24.417712 5722 tablet_service.cc:1467] Processing CreateTablet for tablet 3f25337b712d4928853453e82b017e0b (DEFAULT_TABLE table=test-table [id=7c64173a5ac444349886875255593ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:30:24.417943 5722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f25337b712d4928853453e82b017e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:30:24.419286 5680 tablet_bootstrap.cc:492] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Bootstrap starting.
I20260619 20:30:24.419624 5680 tablet_bootstrap.cc:654] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Neither blocks nor log segments found. Creating new log.
I20260619 20:30:24.420156 5680 tablet_bootstrap.cc:492] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: No bootstrap required, opened a new log
I20260619 20:30:24.420195 5680 ts_tablet_manager.cc:1397] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:24.420308 5680 raft_consensus.cc:348] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:24.420379 5680 raft_consensus.cc:374] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:30:24.420467 5680 raft_consensus.cc:729] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c11f448362e4d50be4f3b13c941cb48, State: Initialized, Role: FOLLOWER
I20260619 20:30:24.420542 5680 consensus_queue.cc:260] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:24.420641 5680 ts_tablet_manager.cc:1428] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 20:30:24.422513 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.417699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.21.151.3:39347)
0619 20:30:24.417862 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.417866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.417876 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:30:24.417910 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.417912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.417928 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.418012 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:24.418142 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.419219 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.421433 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.422160 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.422231 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.422285 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.422383 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.422444 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260619 20:30:24.422513 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.417634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.21.151.2:45021)
0619 20:30:24.417852 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.417857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.417868 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:30:24.417910 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.417912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.417937 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.418143 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:24.418231 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.419275 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.421475 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.422162 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.422231 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.422285 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.422383 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.422437 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
I20260619 20:30:24.422510 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.417437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:45021 (local address 127.0.0.1:44662)
0619 20:30:24.417485 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.417494 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:24.417529 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:24.418036 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:24.418037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:24.418039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:24.418075 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.418077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.419359 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.419360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.420273 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.420276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.422238 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.422239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.422305 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.422309 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:30:24.422371 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:30:24.422428 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:30:24.422511 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.417506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:39347 (local address 127.0.0.1:50624)
0619 20:30:24.417669 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.417679 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:24.417723 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:24.418032 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:24.418033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:24.418035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:24.418077 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.418079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.419319 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.419320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.421366 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.421368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.422239 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.422239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.422305 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.422309 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:30:24.422371 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:30:24.422429 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":923,"thread_start_us":93,"threads_started":1}
I20260619 20:30:24.423293 5860 tablet_service.cc:1467] Processing CreateTablet for tablet 3f25337b712d4928853453e82b017e0b (DEFAULT_TABLE table=test-table [id=7c64173a5ac444349886875255593ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:30:24.423671 5860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f25337b712d4928853453e82b017e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:30:24.426355 6068 tablet_bootstrap.cc:492] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: Bootstrap starting.
I20260619 20:30:24.426651 5992 tablet_service.cc:1467] Processing CreateTablet for tablet 3f25337b712d4928853453e82b017e0b (DEFAULT_TABLE table=test-table [id=7c64173a5ac444349886875255593ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:30:24.426929 5992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f25337b712d4928853453e82b017e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:30:24.426949 6068 tablet_bootstrap.cc:654] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: Neither blocks nor log segments found. Creating new log.
I20260619 20:30:24.427443 6068 log.cc:824] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: Log is configured to *not* fsync() on all Append() calls
I20260619 20:30:24.428117 6068 tablet_bootstrap.cc:492] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: No bootstrap required, opened a new log
I20260619 20:30:24.428194 6068 ts_tablet_manager.cc:1397] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:30:24.429823 6070 tablet_bootstrap.cc:492] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: Bootstrap starting.
I20260619 20:30:24.429752 6068 raft_consensus.cc:348] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:24.429957 6068 raft_consensus.cc:374] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:30:24.429998 6068 raft_consensus.cc:729] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed9b38a21bbf41c68e673e6acedf31d8, State: Initialized, Role: FOLLOWER
I20260619 20:30:24.430104 6068 consensus_queue.cc:260] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:24.430371 5926 heartbeater.cc:503] Master 127.21.151.62:37727 was elected leader, sending a full tablet report...
I20260619 20:30:24.430356 6068 ts_tablet_manager.cc:1428] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:30:24.430442 6070 tablet_bootstrap.cc:654] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: Neither blocks nor log segments found. Creating new log.
I20260619 20:30:24.430991 6070 log.cc:824] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: Log is configured to *not* fsync() on all Append() calls
I20260619 20:30:24.431722 6070 tablet_bootstrap.cc:492] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: No bootstrap required, opened a new log
I20260619 20:30:24.431802 6070 ts_tablet_manager.cc:1397] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:30:24.433068 6070 raft_consensus.cc:348] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:24.433188 6070 raft_consensus.cc:374] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:30:24.433210 6070 raft_consensus.cc:729] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474c4f7cd657462985769a8cde2463c5, State: Initialized, Role: FOLLOWER
I20260619 20:30:24.433346 6070 consensus_queue.cc:260] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:24.433629 6070 ts_tablet_manager.cc:1428] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:30:24.433646 6058 heartbeater.cc:503] Master 127.21.151.62:37727 was elected leader, sending a full tablet report...
I20260619 20:30:24.445634 6073 raft_consensus.cc:482] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:30:24.445713 6073 raft_consensus.cc:504] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:24.446053 6073 leader_election.cc:288] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021), 9c11f448362e4d50be4f3b13c941cb48 (127.21.151.1:37587)
I20260619 20:30:24.447494 6071 raft_consensus.cc:482] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:30:24.447582 6071 raft_consensus.cc:504] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:24.448357 6071 leader_election.cc:288] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347), 9c11f448362e4d50be4f3b13c941cb48 (127.21.151.1:37587)
I20260619 20:30:24.449770 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.446202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37587 (local address 127.21.151.3:42231)
0619 20:30:24.446264 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.446276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:24.446320 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:24.446467 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:24.446468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:24.446470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:24.446533 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.446534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.447513 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.447514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.448797 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.448799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.449540 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.449541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.449620 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.449623 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:30:24.449667 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:30:24.449704 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":54}
I20260619 20:30:24.449824 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.446369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:50281 (local address 127.21.151.2:45021)
0619 20:30:24.446420 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.446422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.446463 (+ 41us) server_negotiation.cc:407] Connection header received
0619 20:30:24.446535 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.446536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.446544 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.446596 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:24.446778 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.447460 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.448918 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.449471 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.449514 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.449520 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.449711 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.449765 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:30:24.450030 6074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.446253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:45021 (local address 127.21.151.3:50281)
0619 20:30:24.446407 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.446416 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:24.446476 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:24.446612 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:24.446613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:24.446615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:24.446674 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.446675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.447543 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.447544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.448803 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.448806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.449574 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.449574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.449641 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.449644 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:30:24.449922 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 20:30:24.449960 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260619 20:30:24.450233 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.446301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:42231 (local address 127.21.151.1:37587)
0619 20:30:24.446341 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.446343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.446356 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:30:24.446385 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.446386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.446395 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.446463 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:24.446636 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.447442 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.448886 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.449438 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.449502 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.449511 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.450130 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.450172 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:30:24.450469 5880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f25337b712d4928853453e82b017e0b" candidate_uuid: "474c4f7cd657462985769a8cde2463c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" is_pre_election: true
I20260619 20:30:24.450469 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f25337b712d4928853453e82b017e0b" candidate_uuid: "474c4f7cd657462985769a8cde2463c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c11f448362e4d50be4f3b13c941cb48" is_pre_election: true
I20260619 20:30:24.450563 5880 raft_consensus.cc:2454] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 474c4f7cd657462985769a8cde2463c5 in term 0.
I20260619 20:30:24.450619 5742 raft_consensus.cc:2454] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 474c4f7cd657462985769a8cde2463c5 in term 0.
I20260619 20:30:24.450724 5945 leader_election.cc:302] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 474c4f7cd657462985769a8cde2463c5, ed9b38a21bbf41c68e673e6acedf31d8; no voters:
I20260619 20:30:24.450835 6073 raft_consensus.cc:2781] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:30:24.450892 6073 raft_consensus.cc:482] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:30:24.450933 6073 raft_consensus.cc:3037] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:30:24.451637 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.448049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37587 (local address 127.21.151.2:38699)
0619 20:30:24.448088 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.448097 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:24.448126 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:24.448524 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:24.448525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:24.448527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:24.448576 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.448578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.449779 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.449779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.450821 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.450823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.451397 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.451398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.451471 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.451474 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:30:24.451524 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:30:24.451561 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 20:30:24.451591 6073 raft_consensus.cc:504] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:24.451699 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.448223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:38699 (local address 127.21.151.1:37587)
0619 20:30:24.448388 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.448390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.448397 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:30:24.448418 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.448419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.448428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.448631 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:24.448662 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.449699 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.450911 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.451305 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.451414 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.451422 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.451551 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.451596 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260619 20:30:24.451721 6073 leader_election.cc:288] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [CANDIDATE]: Term 1 election: Requested vote from peers ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021), 9c11f448362e4d50be4f3b13c941cb48 (127.21.151.1:37587)
I20260619 20:30:24.451841 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f25337b712d4928853453e82b017e0b" candidate_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c11f448362e4d50be4f3b13c941cb48" is_pre_election: true
I20260619 20:30:24.451910 5742 raft_consensus.cc:2454] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed9b38a21bbf41c68e673e6acedf31d8 in term 0.
I20260619 20:30:24.451885 5880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f25337b712d4928853453e82b017e0b" candidate_uuid: "474c4f7cd657462985769a8cde2463c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed9b38a21bbf41c68e673e6acedf31d8"
I20260619 20:30:24.451953 5880 raft_consensus.cc:3037] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:30:24.452586 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.448060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:39347 (local address 127.21.151.2:50921)
0619 20:30:24.448506 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.448516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:24.448586 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:24.448724 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:24.448725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:24.448726 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:30:24.448786 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.448788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.449586 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.449587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.451757 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:24.451760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.452427 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.452427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:24.452475 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.452477 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:30:24.452514 (+ 37us) client_negotiation.cc:240] Negotiation successful
0619 20:30:24.452538 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":75,"threads_started":1}
I20260619 20:30:24.452618 5880 raft_consensus.cc:2454] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 474c4f7cd657462985769a8cde2463c5 in term 1.
I20260619 20:30:24.452651 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.448217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:50921 (local address 127.21.151.3:39347)
0619 20:30:24.448260 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.448261 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.448563 (+ 302us) server_negotiation.cc:407] Connection header received
0619 20:30:24.448645 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.448646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.448653 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.448677 (+ 24us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:30:24.448860 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.449505 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.451854 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.452342 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.452377 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.452382 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.452549 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.452591 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:30:24.452771 5815 leader_election.cc:302] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c11f448362e4d50be4f3b13c941cb48, ed9b38a21bbf41c68e673e6acedf31d8; no voters:
I20260619 20:30:24.452901 5945 leader_election.cc:302] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 474c4f7cd657462985769a8cde2463c5, ed9b38a21bbf41c68e673e6acedf31d8; no voters:
I20260619 20:30:24.452963 6071 raft_consensus.cc:2741] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 20:30:24.453027 6073 raft_consensus.cc:2781] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:30:24.452997 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f25337b712d4928853453e82b017e0b" candidate_uuid: "474c4f7cd657462985769a8cde2463c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c11f448362e4d50be4f3b13c941cb48"
I20260619 20:30:24.453078 5742 raft_consensus.cc:3037] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:30:24.453187 6073 raft_consensus.cc:686] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 1 LEADER]: Becoming Leader. State: Replica: 474c4f7cd657462985769a8cde2463c5, State: Running, Role: LEADER
I20260619 20:30:24.452992 6012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f25337b712d4928853453e82b017e0b" candidate_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474c4f7cd657462985769a8cde2463c5" is_pre_election: true
I20260619 20:30:24.453531 6073 consensus_queue.cc:237] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:24.453912 5742 raft_consensus.cc:2454] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 474c4f7cd657462985769a8cde2463c5 in term 1.
I20260619 20:30:24.454340 5599 catalog_manager.cc:5495] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 474c4f7cd657462985769a8cde2463c5 (127.21.151.3). New cstate: current_term: 1 leader_uuid: "474c4f7cd657462985769a8cde2463c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } health_report { overall_health: UNKNOWN } } }
I20260619 20:30:24.467211 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.465123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.21.151.3:39347)
0619 20:30:24.465158 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:24.465160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:24.465175 (+ 15us) server_negotiation.cc:407] Connection header received
0619 20:30:24.465248 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:24.465248 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:24.465261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:24.465312 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:30:24.465432 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.465835 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.466560 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:24.466729 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:24.466780 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:24.466906 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:30:24.466994 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:30:24.467051 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:24.467086 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 20:30:24.467100 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:30:24.468879 5880 raft_consensus.cc:1261] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 1 FOLLOWER]: Refusing update from remote peer 474c4f7cd657462985769a8cde2463c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:30:24.468885 5742 raft_consensus.cc:1261] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Refusing update from remote peer 474c4f7cd657462985769a8cde2463c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:30:24.469188 6077 consensus_queue.cc:1035] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:30:24.469307 6073 consensus_queue.cc:1035] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:30:24.471580 6081 mvcc.cc:205] Tried to move back new op lower bound from 7298666596222042112 to 7298666596162465792. Current Snapshot: MvccSnapshot[applied={T|T < 7298666596222042112}]
I20260619 20:30:24.472003 6082 mvcc.cc:205] Tried to move back new op lower bound from 7298666596222042112 to 7298666596162465792. Current Snapshot: MvccSnapshot[applied={T|T < 7298666596222042112}]
I20260619 20:30:24.474033 5600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:30:24.474143 5600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:30:24.474874 5600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Sending DeleteTablet for 3 replicas of tablet 3f25337b712d4928853453e82b017e0b
I20260619 20:30:24.475209 5722 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f25337b712d4928853453e82b017e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:30:24 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:49458
I20260619 20:30:24.475239 5992 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f25337b712d4928853453e82b017e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:30:24 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:50624
I20260619 20:30:24.475417 5797 tablet_replica.cc:330] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: stopping tablet replica
I20260619 20:30:24.475433 6090 tablet_replica.cc:330] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: stopping tablet replica
I20260619 20:30:24.475485 5797 raft_consensus.cc:2229] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:30:24.475538 5797 raft_consensus.cc:2258] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:30:24.475540 6090 raft_consensus.cc:2229] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:30:24.475673 6090 raft_consensus.cc:2258] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:30:24.475863 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 5638
I20260619 20:30:24.475907 5860 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f25337b712d4928853453e82b017e0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:30:24 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:44662
I20260619 20:30:24.475998 5797 ts_tablet_manager.cc:1905] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:30:24.479817 6091 tablet_replica.cc:330] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: stopping tablet replica
I20260619 20:30:24.479924 6091 raft_consensus.cc:2229] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:30:24.480026 6091 raft_consensus.cc:2258] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:30:24.480134 6090 ts_tablet_manager.cc:1905] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:30:24.480420 6091 ts_tablet_manager.cc:1905] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 20:30:24.482132 5586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:37587 (error 108)
I20260619 20:30:24.482388 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 5793
I20260619 20:30:24.482589 6090 ts_tablet_manager.cc:1918] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:30:24.482714 6090 log.cc:1196] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/3f25337b712d4928853453e82b017e0b
I20260619 20:30:24.483115 6090 ts_tablet_manager.cc:1939] T 3f25337b712d4928853453e82b017e0b P 474c4f7cd657462985769a8cde2463c5: Deleting consensus metadata
I20260619 20:30:24.483562 5587 catalog_manager.cc:4841] TS 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347): tablet 3f25337b712d4928853453e82b017e0b (table test-table [id=7c64173a5ac444349886875255593ad3]) successfully deleted
I20260619 20:30:24.484479 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.484349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:37587 (local address 127.0.0.1:49474)
0619 20:30:24.484406 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.484442 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:37587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 20:30:24.484606 5586 catalog_manager.cc:4584] TS 9c11f448362e4d50be4f3b13c941cb48 (127.21.151.1:37587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f25337b712d4928853453e82b017e0b: Network error: Client connection negotiation failed: client connection to 127.21.151.1:37587: connect: Connection refused (error 111)
I20260619 20:30:24.488337 6064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:24.488217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:45021 (local address 127.0.0.1:44676)
0619 20:30:24.488266 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:24.488291 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.2:45021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 20:30:24.488411 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 5929
I20260619 20:30:24.493134 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 5567
I20260619 20:30:44.498512 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:37727
--webserver_interface=127.21.151.62
--webserver_port=36427
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:37727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:30:44.578778 6093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:44.578984 6093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:44.579006 6093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:44.580241 6093 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 20:30:44.580278 6093 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 20:30:44.580291 6093 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 20:30:44.580303 6093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:30:44.580334 6093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:44.580483 6093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:44.580538 6093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:30:44.580550 6093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:30:44.582748 6093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:37727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=36427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:44.583003 6093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:44.583232 6093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:30:44.588495 6099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:44.588486 6101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:44.588486 6098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:44.588583 6093 server_base.cc:1034] running on GCE node
I20260619 20:30:44.588889 6093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:44.589134 6093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:44.590276 6093 hybrid_clock.cc:648] HybridClock initialized: now 1781901044590263 us; error 31 us; skew 500 ppm
Jun 19 20:30:44 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901044, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:44.596975 6093 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:30:44.597419 6093 webserver.cc:460] Webserver started at http://127.21.151.62:36427/ using document root <none> and password file <none>
I20260619 20:30:44.597659 6093 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:44.597703 6093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:44.599028 6093 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:44.599720 6108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:44.599912 6093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:44.600026 6093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "aef803797e4a45d4a796c208ce99c58d"
format_stamp: "Formatted at 2026-06-19 20:30:02 on dist-test-slave-f9rk"
server_key: "0c4f990080a18777632ef92149c283c6"
server_key_iv: "85b1becc56a19f32261e795cd7c35856"
server_key_version: "encryptionkey@0"
I20260619 20:30:44.600374 6093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:44.613970 6093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:44.616129 6093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:44.616284 6093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:44.620077 6093 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:37727
I20260619 20:30:44.620112 6160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:37727 every 8 connection(s)
I20260619 20:30:44.620503 6093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:30:44.621949 6161 sys_catalog.cc:263] Verifying existing consensus state
I20260619 20:30:44.622522 6161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Bootstrap starting.
I20260619 20:30:44.623487 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 6093
I20260619 20:30:44.623821 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:37587
--local_ip_for_outbound_sockets=127.21.151.1
--tserver_master_addrs=127.21.151.62:37727
--webserver_port=46521
--webserver_interface=127.21.151.1
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:30:44.631479 6161 log.cc:824] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:30:44.633704 6161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:30:44.634079 6161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Bootstrap complete.
I20260619 20:30:44.636363 6161 raft_consensus.cc:348] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:44.636577 6161 raft_consensus.cc:729] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef803797e4a45d4a796c208ce99c58d, State: Initialized, Role: FOLLOWER
I20260619 20:30:44.636755 6161 consensus_queue.cc:260] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:44.636812 6161 raft_consensus.cc:388] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:30:44.636886 6161 raft_consensus.cc:482] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:30:44.636938 6161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 2 FOLLOWER]: Advancing to term 3
I20260619 20:30:44.637893 6161 raft_consensus.cc:504] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:44.637984 6161 leader_election.cc:302] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aef803797e4a45d4a796c208ce99c58d; no voters:
I20260619 20:30:44.638226 6161 leader_election.cc:288] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 20:30:44.638259 6165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 3 FOLLOWER]: Leader election won for term 3
I20260619 20:30:44.638393 6165 raft_consensus.cc:686] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [term 3 LEADER]: Becoming Leader. State: Replica: aef803797e4a45d4a796c208ce99c58d, State: Running, Role: LEADER
I20260619 20:30:44.638486 6161 sys_catalog.cc:564] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:30:44.638479 6165 consensus_queue.cc:237] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } }
I20260619 20:30:44.639333 6166 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aef803797e4a45d4a796c208ce99c58d. Latest consensus state: current_term: 3 leader_uuid: "aef803797e4a45d4a796c208ce99c58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } } }
I20260619 20:30:44.639430 6166 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: This master's current role is: LEADER
I20260619 20:30:44.639566 6167 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "aef803797e4a45d4a796c208ce99c58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef803797e4a45d4a796c208ce99c58d" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 37727 } } }
I20260619 20:30:44.639628 6175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:30:44.640483 6175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:30:44.640573 6175 catalog_manager.cc:1259] Loaded cluster ID: ea3cde07413c4606b1d4cca9f03724d7
I20260619 20:30:44.640594 6175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:30:44.640661 6167 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d [sys.catalog]: This master's current role is: LEADER
I20260619 20:30:44.642056 6175 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:30:44.642231 6175 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Loaded TSK: 0
I20260619 20:30:44.642266 6175 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aef803797e4a45d4a796c208ce99c58d: Loaded TSK: 1
F20260619 20:30:44.642499 6175 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fde86c7d9ad google::LogMessage::Fail() at ??:0
@ 0x7fde86c8088a google::LogMessage::SendToLog() at ??:0
@ 0x7fde86c7d4de google::LogMessage::Flush() at ??:0
@ 0x7fde86c7ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fde893ea104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fde893e7d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fde8ebdb1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fde8ebdb48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fde8ebdb594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fde8eba1c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fde8ebabad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fde88b4b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fde88b40954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fde899a46db start_thread at ??:0
@ 0x7fde8577b61f clone at ??:0
W20260619 20:30:44.725991 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:44.726191 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:44.726215 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:44.727368 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:44.727412 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:44.727445 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:30:44.728947 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:37587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=46521
--tserver_master_addrs=127.21.151.62:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:44.729187 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:44.729434 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:30:44.734421 6203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:44.734385 6202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:44.734375 6205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:44.735059 6163 server_base.cc:1034] running on GCE node
I20260619 20:30:44.735304 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:44.735543 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:44.736702 6163 hybrid_clock.cc:648] HybridClock initialized: now 1781901044736690 us; error 32 us; skew 500 ppm
Jun 19 20:30:44 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901044, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:44.739420 6163 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:30:44.739993 6163 webserver.cc:460] Webserver started at http://127.21.151.1:46521/ using document root <none> and password file <none>
I20260619 20:30:44.740291 6163 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:44.740367 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:44.742213 6163 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:30:44.743140 6212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:44.743345 6163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:44.743427 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "9c11f448362e4d50be4f3b13c941cb48"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "c7b61d50aa01f26f9c6e1732d33f489b"
server_key_iv: "bb7ae5edee3fc5040591c60d774665e4"
server_key_version: "encryptionkey@0"
I20260619 20:30:44.743867 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:44.765125 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:44.767550 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:44.767758 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:44.768014 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:30:44.768499 6219 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 20:30:44.770696 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 20:30:44.770762 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 20:30:44.770805 6163 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 20:30:44.771437 6163 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 20:30:44.771495 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 20:30:44.771572 6219 tablet_bootstrap.cc:492] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Bootstrap starting.
I20260619 20:30:44.773622 6219 log.cc:824] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Log is configured to *not* fsync() on all Append() calls
I20260619 20:30:44.777134 6219 tablet_bootstrap.cc:492] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 20:30:44.777580 6219 tablet_bootstrap.cc:492] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Bootstrap complete.
I20260619 20:30:44.777815 6219 ts_tablet_manager.cc:1397] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 20:30:44.778661 6163 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:37587
I20260619 20:30:44.778815 6327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:37587 every 8 connection(s)
I20260619 20:30:44.779052 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:30:44.779383 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 6163
I20260619 20:30:44.779750 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:45021
--local_ip_for_outbound_sockets=127.21.151.2
--tserver_master_addrs=127.21.151.62:37727
--webserver_port=42065
--webserver_interface=127.21.151.2
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:30:44.779956 6219 raft_consensus.cc:348] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:44.780273 6219 raft_consensus.cc:729] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c11f448362e4d50be4f3b13c941cb48, State: Initialized, Role: FOLLOWER
I20260619 20:30:44.780458 6219 consensus_queue.cc:260] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:44.780736 6219 ts_tablet_manager.cc:1428] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260619 20:30:44.838480 6331 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:30:44.780524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:33969)
0619 20:30:44.781109 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:44.781154 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:44.781301 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:44.838254 (+ 56953us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":92,"threads_started":1}
W20260619 20:30:44.838747 6328 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:37727 (0 consecutive failures): Network error: Failed to ping master at 127.21.151.62:37727: Client connection negotiation failed: client connection to 127.21.151.62:37727: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 20:30:44.839104 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:44.838978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:45861)
0619 20:30:44.839034 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:44.839082 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:30:44.839401 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:44.839311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:45183)
0619 20:30:44.839355 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:44.839377 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 20:30:44.839501 6328 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 20:30:44.863919 6332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:44.864152 6332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:44.864198 6332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:44.865486 6332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:44.865549 6332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:44.865595 6332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:30:44.867067 6332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:45021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=42065
--tserver_master_addrs=127.21.151.62:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:44.867340 6332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:44.867580 6332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:30:44.872500 6340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:44.872520 6337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:44.872512 6338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:44.872972 6332 server_base.cc:1034] running on GCE node
I20260619 20:30:44.873185 6332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:44.873438 6332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:44.874603 6332 hybrid_clock.cc:648] HybridClock initialized: now 1781901044874600 us; error 34 us; skew 500 ppm
Jun 19 20:30:44 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901044, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:44.877209 6332 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:30:44.877604 6332 webserver.cc:460] Webserver started at http://127.21.151.2:42065/ using document root <none> and password file <none>
I20260619 20:30:44.877813 6332 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:44.877862 6332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:44.879151 6332 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:44.879813 6347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:44.879967 6332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:30:44.880054 6332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "ed9b38a21bbf41c68e673e6acedf31d8"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "7f2bb9df2b4640cf44b791cbb96416c3"
server_key_iv: "ef43aec1b49867aab681180f5786c6f5"
server_key_version: "encryptionkey@0"
I20260619 20:30:44.880455 6332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:44.905002 6332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:44.906963 6332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:44.907079 6332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:44.907275 6332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:30:44.907902 6354 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 20:30:44.909322 6354 ts_tablet_manager.cc:1852] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 20:30:44.909633 6354 ts_tablet_manager.cc:1905] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:30:44.911913 6354 ts_tablet_manager.cc:1918] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:30:44.912040 6354 log.cc:1196] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/3f25337b712d4928853453e82b017e0b
I20260619 20:30:44.912304 6354 ts_tablet_manager.cc:1939] T 3f25337b712d4928853453e82b017e0b P ed9b38a21bbf41c68e673e6acedf31d8: Deleting consensus metadata
I20260619 20:30:44.912545 6332 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 20:30:44.912595 6332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 20:30:44.912642 6332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:30:44.912674 6332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:44.918581 6332 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:45021
I20260619 20:30:44.918684 6461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:45021 every 8 connection(s)
I20260619 20:30:44.919025 6332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:30:44.920176 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:44.919815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:36749)
0619 20:30:44.919973 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:44.920046 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
W20260619 20:30:44.920410 6462 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:37727 (0 consecutive failures): Network error: Failed to ping master at 127.21.151.62:37727: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
I20260619 20:30:44.920710 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:44.920624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:37557)
0619 20:30:44.920669 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:44.920695 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:30:44.920954 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:44.920908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:56509)
0619 20:30:44.920926 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:44.920943 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260619 20:30:44.921052 6462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:30:44.925964 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 6332
I20260619 20:30:44.926234 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:39347
--local_ip_for_outbound_sockets=127.21.151.3
--tserver_master_addrs=127.21.151.62:37727
--webserver_port=33341
--webserver_interface=127.21.151.3
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:30:45.007894 6465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:30:45.008122 6465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:30:45.008148 6465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:30:45.009254 6465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:30:45.009289 6465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:30:45.009315 6465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:30:45.010782 6465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:39347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=33341
--tserver_master_addrs=127.21.151.62:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:30:45.010967 6465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:30:45.011140 6465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:30:45.016400 6465 server_base.cc:1034] running on GCE node
W20260619 20:30:45.016288 6470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:45.016259 6471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:30:45.016258 6473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:30:45.016851 6465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:30:45.017063 6465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:30:45.018199 6465 hybrid_clock.cc:648] HybridClock initialized: now 1781901045018184 us; error 34 us; skew 500 ppm
Jun 19 20:30:45 dist-test-slave-f9rk krb5kdc[5018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901045, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:30:45.020735 6465 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:30:45.021170 6465 webserver.cc:460] Webserver started at http://127.21.151.3:33341/ using document root <none> and password file <none>
I20260619 20:30:45.021399 6465 fs_manager.cc:359] Metadata directory not provided
I20260619 20:30:45.021476 6465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:30:45.022778 6465 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:30:45.023331 6480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:45.023520 6465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:30:45.023615 6465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "474c4f7cd657462985769a8cde2463c5"
format_stamp: "Formatted at 2026-06-19 20:30:03 on dist-test-slave-f9rk"
server_key: "d067b2f0eaef9b61cc7becf3994417ac"
server_key_iv: "5782dcb4c7a879cdca2a663b58e447db"
server_key_version: "encryptionkey@0"
I20260619 20:30:45.024022 6465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:30:45.046247 6465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:30:45.048377 6465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:30:45.048553 6465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:30:45.048794 6465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:30:45.049106 6465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:30:45.049162 6465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:45.049203 6465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:30:45.049228 6465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:30:45.054847 6465 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:39347
I20260619 20:30:45.054908 6593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:39347 every 8 connection(s)
I20260619 20:30:45.055276 6465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:30:45.056373 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:45.056005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:58367)
0619 20:30:45.056162 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:45.056233 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260619 20:30:45.056860 6594 heartbeater.cc:644] Failed to heartbeat to 127.21.151.62:37727 (0 consecutive failures): Network error: Failed to ping master at 127.21.151.62:37727: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
I20260619 20:30:45.057158 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:45.057069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:58127)
0619 20:30:45.057117 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:45.057142 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 20:30:45.057479 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:45.057397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:34897)
0619 20:30:45.057445 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:45.057466 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 20:30:45.057617 6594 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 20:30:45.061465 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 6465
I20260619 20:30:45.840111 6598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:45.839888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:40375)
0619 20:30:45.840010 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:45.840047 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260619 20:30:45.921556 6599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:45.921370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:42111)
0619 20:30:45.921488 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:45.921521 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260619 20:30:46.058172 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:46.057976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:39815)
0619 20:30:46.058088 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:46.058123 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260619 20:30:46.554625 6601 raft_consensus.cc:482] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:30:46.554744 6601 raft_consensus.cc:504] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:46.555029 6601 leader_election.cc:288] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021), 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347)
Jun 19 20:30:46 dist-test-slave-f9rk krb5kdc[5018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901044, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.3@KRBTEST.COM
Jun 19 20:30:46 dist-test-slave-f9rk krb5kdc[5018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901044, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.2@KRBTEST.COM
I20260619 20:30:46.565749 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:46.555338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:53867 (local address 127.21.151.2:45021)
0619 20:30:46.555506 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:46.555523 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:46.555541 (+ 18us) server_negotiation.cc:407] Connection header received
0619 20:30:46.556071 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:46.556074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:46.556112 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:46.556261 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:46.557247 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:46.558090 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:46.558967 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:46.559372 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:46.559497 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:46.562456 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:46.562460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:46.562460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:46.562470 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:46.564318 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:46.564661 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:46.564664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:46.564665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:46.564753 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:46.564961 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:46.564962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:46.564962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:46.565222 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:46.565303 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:46.565508 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 20:30:46.565630 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":47,"threads_started":1}
I20260619 20:30:46.565749 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:46.555207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:39347 (local address 127.21.151.1:34261)
0619 20:30:46.555381 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:46.555392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:46.555459 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:46.556250 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:46.556252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:46.556736 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:30:46.556961 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:46.556967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:46.558057 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:46.558058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:46.558872 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:46.558875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:46.559481 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:46.559481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:46.559649 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:46.560384 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:30:46.560395 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:30:46.561940 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:30:46.564167 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:46.564173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:46.564175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:46.564401 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:46.564679 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:46.564679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:46.564680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:46.564784 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:46.565188 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:30:46.565194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:30:46.565339 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 20:30:46.565462 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 20:30:46.565642 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":12,"thread_start_us":46,"threads_started":1}
I20260619 20:30:46.565781 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:46.555319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:34261 (local address 127.21.151.3:39347)
0619 20:30:46.555479 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 20:30:46.555493 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 20:30:46.555524 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:30:46.556021 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:30:46.556023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:30:46.556068 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:30:46.556219 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:30:46.557090 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:46.557952 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:46.558994 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:30:46.559378 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:30:46.559502 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:46.562048 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:30:46.562052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:30:46.562052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:30:46.562063 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:30:46.564088 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:46.564513 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:46.564516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:46.564516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:46.564611 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:30:46.564860 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:30:46.564861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:30:46.564862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:30:46.565098 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:30:46.565192 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 20:30:46.565526 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 20:30:46.565668 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":53,"threads_started":1}
I20260619 20:30:46.565856 6602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:46.555207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:45021 (local address 127.21.151.1:53867)
0619 20:30:46.555314 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:46.555328 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:30:46.555389 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:30:46.556260 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:30:46.556261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:30:46.556737 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:30:46.557133 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:46.557136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:46.558194 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:46.558195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:46.558871 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:30:46.558873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:30:46.559483 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:30:46.559484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:30:46.559605 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:30:46.560349 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:30:46.560358 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:30:46.562325 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:30:46.564398 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:46.564399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:46.564399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:46.564573 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:46.564828 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:30:46.564829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:30:46.564829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:30:46.564887 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:30:46.565294 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:30:46.565294 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:30:46.565337 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 20:30:46.565455 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 20:30:46.565716 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":136,"thread_start_us":58,"threads_started":1}
W20260619 20:30:46.566804 6216 leader_election.cc:341] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347): Not found: Tablet not found: 3f25337b712d4928853453e82b017e0b
W20260619 20:30:46.567180 6213 leader_election.cc:341] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021): Not found: Tablet not found: 3f25337b712d4928853453e82b017e0b
I20260619 20:30:46.567226 6213 leader_election.cc:302] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c11f448362e4d50be4f3b13c941cb48; no voters: 474c4f7cd657462985769a8cde2463c5, ed9b38a21bbf41c68e673e6acedf31d8
I20260619 20:30:46.567323 6601 raft_consensus.cc:2726] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:30:46.840724 6602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:46.840586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:44489)
0619 20:30:46.840635 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:46.840682 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 20:30:46.922210 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:46.921988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:37217)
0619 20:30:46.922130 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:46.922162 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260619 20:30:47.058847 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:47.058620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:42537)
0619 20:30:47.058757 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:47.058796 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260619 20:30:47.841359 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:47.841139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:35003)
0619 20:30:47.841280 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:47.841315 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260619 20:30:47.922899 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:47.922671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:56961)
0619 20:30:47.922807 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:47.922859 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260619 20:30:48.059486 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:48.059308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:58009)
0619 20:30:48.059418 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:48.059447 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260619 20:30:48.299916 6611 raft_consensus.cc:482] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:30:48.300028 6611 raft_consensus.cc:504] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:48.300203 6611 leader_election.cc:288] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021), 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347)
W20260619 20:30:48.300632 6213 leader_election.cc:341] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021): Not found: Tablet not found: 3f25337b712d4928853453e82b017e0b
W20260619 20:30:48.300716 6216 leader_election.cc:341] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347): Not found: Tablet not found: 3f25337b712d4928853453e82b017e0b
I20260619 20:30:48.300834 6216 leader_election.cc:302] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c11f448362e4d50be4f3b13c941cb48; no voters: 474c4f7cd657462985769a8cde2463c5, ed9b38a21bbf41c68e673e6acedf31d8
I20260619 20:30:48.300931 6611 raft_consensus.cc:2726] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:30:48.842017 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:48.841795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:40927)
0619 20:30:48.841930 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:48.841969 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260619 20:30:48.923557 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:48.923356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:42857)
0619 20:30:48.923473 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:48.923502 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260619 20:30:49.060200 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:49.059974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:53289)
0619 20:30:49.060109 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:49.060146 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260619 20:30:49.842690 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:49.842493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:51857)
0619 20:30:49.842612 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:49.842645 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260619 20:30:49.906209 6616 raft_consensus.cc:482] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:30:49.906311 6616 raft_consensus.cc:504] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:49.906522 6616 leader_election.cc:288] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021), 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347)
W20260619 20:30:49.906961 6216 leader_election.cc:341] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347): Not found: Tablet not found: 3f25337b712d4928853453e82b017e0b
W20260619 20:30:49.907011 6213 leader_election.cc:341] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021): Not found: Tablet not found: 3f25337b712d4928853453e82b017e0b
I20260619 20:30:49.907037 6216 leader_election.cc:302] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c11f448362e4d50be4f3b13c941cb48; no voters: 474c4f7cd657462985769a8cde2463c5, ed9b38a21bbf41c68e673e6acedf31d8
I20260619 20:30:49.907107 6616 raft_consensus.cc:2726] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:30:49.924256 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:49.924043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:36471)
0619 20:30:49.924158 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:49.924198 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260619 20:30:50.060849 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:50.060653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:39275)
0619 20:30:50.060769 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:50.060809 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260619 20:30:50.843348 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:50.843145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:50727)
0619 20:30:50.843270 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:50.843302 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260619 20:30:50.924984 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:50.924732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:35227)
0619 20:30:50.924891 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:50.924937 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260619 20:30:51.061524 6621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:51.061308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:39151)
0619 20:30:51.061439 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:51.061482 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260619 20:30:51.844177 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:51.843933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:57559)
0619 20:30:51.844086 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:51.844130 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260619 20:30:51.925709 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:51.925485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:33323)
0619 20:30:51.925627 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:51.925666 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260619 20:30:52.062216 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:52.061984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:45917)
0619 20:30:52.062130 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:52.062171 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260619 20:30:52.226085 6625 raft_consensus.cc:482] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:30:52.226179 6625 raft_consensus.cc:504] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:52.226447 6625 leader_election.cc:288] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021), 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347)
W20260619 20:30:52.226890 6213 leader_election.cc:341] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021): Not found: Tablet not found: 3f25337b712d4928853453e82b017e0b
W20260619 20:30:52.226945 6216 leader_election.cc:341] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347): Not found: Tablet not found: 3f25337b712d4928853453e82b017e0b
I20260619 20:30:52.226977 6216 leader_election.cc:302] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c11f448362e4d50be4f3b13c941cb48; no voters: 474c4f7cd657462985769a8cde2463c5, ed9b38a21bbf41c68e673e6acedf31d8
I20260619 20:30:52.227068 6625 raft_consensus.cc:2726] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:30:52.844949 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:52.844703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:48785)
0619 20:30:52.844854 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:52.844897 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260619 20:30:52.926484 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:52.926191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:60863)
0619 20:30:52.926402 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:52.926441 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":59,"threads_started":1}
I20260619 20:30:53.062920 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:53.062691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:59307)
0619 20:30:53.062833 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:53.062872 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260619 20:30:53.845829 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:53.845584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:55511)
0619 20:30:53.845741 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:53.845781 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260619 20:30:53.927196 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:53.926937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:50657)
0619 20:30:53.927119 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:53.927155 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":60,"threads_started":1}
I20260619 20:30:54.063570 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:54.063385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:34281)
0619 20:30:54.063498 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:54.063532 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260619 20:30:54.846766 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:54.846506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:39155)
0619 20:30:54.846637 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:54.846721 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260619 20:30:54.927948 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:54.927725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:58753)
0619 20:30:54.927861 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:54.927903 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260619 20:30:55.064296 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:55.064070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:45847)
0619 20:30:55.064213 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:55.064253 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260619 20:30:55.847580 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:55.847301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:58495)
0619 20:30:55.847496 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:55.847534 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":109,"threads_started":1}
I20260619 20:30:55.928596 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:55.928391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:42379)
0619 20:30:55.928519 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:55.928553 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260619 20:30:56.065089 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:56.064890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:38915)
0619 20:30:56.065009 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:56.065045 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260619 20:30:56.718530 6638 raft_consensus.cc:482] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:30:56.718659 6638 raft_consensus.cc:504] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9b38a21bbf41c68e673e6acedf31d8" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 45021 } } peers { permanent_uuid: "474c4f7cd657462985769a8cde2463c5" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39347 } } peers { permanent_uuid: "9c11f448362e4d50be4f3b13c941cb48" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 37587 } }
I20260619 20:30:56.718863 6638 leader_election.cc:288] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021), 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347)
W20260619 20:30:56.719291 6213 leader_election.cc:341] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed9b38a21bbf41c68e673e6acedf31d8 (127.21.151.2:45021): Not found: Tablet not found: 3f25337b712d4928853453e82b017e0b
W20260619 20:30:56.719404 6216 leader_election.cc:341] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 474c4f7cd657462985769a8cde2463c5 (127.21.151.3:39347): Not found: Tablet not found: 3f25337b712d4928853453e82b017e0b
I20260619 20:30:56.719465 6216 leader_election.cc:302] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c11f448362e4d50be4f3b13c941cb48; no voters: 474c4f7cd657462985769a8cde2463c5, ed9b38a21bbf41c68e673e6acedf31d8
I20260619 20:30:56.719539 6638 raft_consensus.cc:2726] T 3f25337b712d4928853453e82b017e0b P 9c11f448362e4d50be4f3b13c941cb48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 20:30:56.848271 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:56.848000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:43695)
0619 20:30:56.848174 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:56.848225 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
I20260619 20:30:56.929239 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:56.929034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:48341)
0619 20:30:56.929163 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:56.929198 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260619 20:30:57.065802 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:57.065535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:56247)
0619 20:30:57.065679 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:57.065742 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260619 20:30:57.848935 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:57.848714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:35411)
0619 20:30:57.848856 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:57.848889 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260619 20:30:57.929906 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:57.929693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:33991)
0619 20:30:57.929820 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:57.929860 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260619 20:30:58.066542 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:58.066274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:46863)
0619 20:30:58.066434 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:58.066497 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260619 20:30:58.849604 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:58.849401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:43205)
0619 20:30:58.849523 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:58.849557 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260619 20:30:58.930569 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:58.930373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:46811)
0619 20:30:58.930497 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:58.930527 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260619 20:30:59.067224 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:59.066965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:32953)
0619 20:30:59.067107 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:59.067154 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260619 20:30:59.850337 6648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:59.850137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.1:38337)
0619 20:30:59.850254 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:59.850292 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260619 20:30:59.931284 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:30:59.931072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.2:37635)
0619 20:30:59.931198 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 20:30:59.931240 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260619 20:31:00.067978 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:00.067773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:37727 (local address 127.21.151.3:52807)
0619 20:31:00.067892 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:00.067935 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.62:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260619 20:31:00.072705 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 6163
I20260619 20:31:00.077255 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 6332
I20260619 20:31:00.081954 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 6465
2026-06-19T20:31:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59352 ms)
[----------] 30 tests from SecurityITest (231657 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:00 dist-test-slave-f9rk krb5kdc[6653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:00 dist-test-slave-f9rk krb5kdc[6653](info): set up 2 sockets
Jun 19 20:31:00 dist-test-slave-f9rk krb5kdc[6653](info): commencing operation
krb5kdc: starting...
W20260619 20:31:01.928550 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:02 dist-test-slave-f9rk krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:02.044754 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:43227
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:43227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:02.125097 6666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:02.125336 6666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:02.125375 6666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:02.126479 6666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:02.126564 6666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:02.126586 6666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:02.126606 6666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:02.126636 6666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:02.128090 6666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:43227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:43227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:02.128367 6666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:02.128587 6666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:02.133299 6672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:02.133469 6666 server_base.cc:1034] running on GCE node
W20260619 20:31:02.133307 6674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:02.133307 6671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:02.133791 6666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:02.134042 6666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:02.135183 6666 hybrid_clock.cc:648] HybridClock initialized: now 1781901062135158 us; error 41 us; skew 500 ppm
Jun 19 20:31:02 dist-test-slave-f9rk krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901062, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:02.137553 6666 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:02.137989 6666 webserver.cc:460] Webserver started at http://127.21.151.62:43571/ using document root <none> and password file <none>
I20260619 20:31:02.138166 6666 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:02.138207 6666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:02.138293 6666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:02.139267 6666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "f3245962e1f341ef858609aef6a51713"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "b671cfbacd4b1e9a6bc81d88fcf1e015"
server_key_iv: "5ef7e2d8a407116e1e7247889f11abe9"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.139571 6666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "f3245962e1f341ef858609aef6a51713"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "b671cfbacd4b1e9a6bc81d88fcf1e015"
server_key_iv: "5ef7e2d8a407116e1e7247889f11abe9"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.140919 6666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:31:02.141547 6681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.141713 6666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.141815 6666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "f3245962e1f341ef858609aef6a51713"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "b671cfbacd4b1e9a6bc81d88fcf1e015"
server_key_iv: "5ef7e2d8a407116e1e7247889f11abe9"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.141894 6666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:02.151409 6666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:02.153159 6666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:02.153313 6666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:02.157277 6666 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:43227
I20260619 20:31:02.157311 6733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:43227 every 8 connection(s)
I20260619 20:31:02.157644 6666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:02.158501 6734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:02.159637 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 6666
I20260619 20:31:02.159742 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:02.159940 22108 external_mini_cluster.cc:1428] Setting key 9c5be590e76134b041e237a2d6dbca3f
I20260619 20:31:02.161360 6734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713: Bootstrap starting.
I20260619 20:31:02.162114 6734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:02.162432 6734 log.cc:824] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:02.163117 6734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713: No bootstrap required, opened a new log
I20260619 20:31:02.164949 6734 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3245962e1f341ef858609aef6a51713" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43227 } }
I20260619 20:31:02.165083 6734 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:02.165105 6734 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3245962e1f341ef858609aef6a51713, State: Initialized, Role: FOLLOWER
I20260619 20:31:02.165271 6734 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3245962e1f341ef858609aef6a51713" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43227 } }
I20260619 20:31:02.165356 6734 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:02.165388 6734 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:02.165455 6734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:02.166209 6734 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3245962e1f341ef858609aef6a51713" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43227 } }
I20260619 20:31:02.166321 6734 leader_election.cc:302] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3245962e1f341ef858609aef6a51713; no voters:
I20260619 20:31:02.166586 6734 leader_election.cc:288] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:02.166603 6739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:02.166759 6739 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [term 1 LEADER]: Becoming Leader. State: Replica: f3245962e1f341ef858609aef6a51713, State: Running, Role: LEADER
I20260619 20:31:02.166893 6734 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:02.166903 6739 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3245962e1f341ef858609aef6a51713" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43227 } }
Jun 19 20:31:02 dist-test-slave-f9rk krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:02.167337 6740 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3245962e1f341ef858609aef6a51713. Latest consensus state: current_term: 1 leader_uuid: "f3245962e1f341ef858609aef6a51713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3245962e1f341ef858609aef6a51713" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43227 } } }
I20260619 20:31:02.167434 6740 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:02.167583 6741 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3245962e1f341ef858609aef6a51713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3245962e1f341ef858609aef6a51713" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43227 } } }
I20260619 20:31:02.167670 6741 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:02.167797 6744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:02.168357 6744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:02.170347 6744 catalog_manager.cc:1347] Generated new cluster ID: a0d033e34bb44e81a56b536c996ef814
I20260619 20:31:02.170426 6744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:02.171634 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.160464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58126 (local address 127.21.151.62:43227)
0619 20:31:02.160628 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.160635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.160660 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:31:02.161155 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.161161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.161206 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.161317 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:02.162094 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.163236 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.164107 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.164506 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.164633 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.167827 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:02.167832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:02.167832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:02.167844 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:02.169909 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:02.170299 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:02.170302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:02.170302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:02.170427 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:02.170650 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:02.170651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:02.170652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:02.170967 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:02.171123 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.171245 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.171388 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260619 20:31:02.177678 6744 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:02.178187 6744 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:02.188278 6744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713: Generated new TSK 0
I20260619 20:31:02.188465 6744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:31:02.214020 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:43227
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:02.295550 6760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:02.295828 6760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:02.295859 6760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:02.297019 6760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:02.297058 6760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:02.297086 6760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:31:02.298560 6760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:43227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:02.298813 6760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:02.299057 6760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:02.304062 6768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:02.304070 6766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:02.304304 6760 server_base.cc:1034] running on GCE node
W20260619 20:31:02.304080 6765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:02.304598 6760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:02.304816 6760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:02.305960 6760 hybrid_clock.cc:648] HybridClock initialized: now 1781901062305948 us; error 30 us; skew 500 ppm
Jun 19 20:31:02 dist-test-slave-f9rk krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901062, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:02.308496 6760 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:31:02.308931 6760 webserver.cc:460] Webserver started at http://127.21.151.1:33063/ using document root <none> and password file <none>
I20260619 20:31:02.309165 6760 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:02.309212 6760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:02.309373 6760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:02.310415 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "0079b7e1912f4abab6b1e62068197aaf"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "ed263a43b3cdd3ffd682afb01de7b884"
server_key_iv: "2301e032be54df862b986094133c476f"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.310781 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "0079b7e1912f4abab6b1e62068197aaf"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "ed263a43b3cdd3ffd682afb01de7b884"
server_key_iv: "2301e032be54df862b986094133c476f"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.312166 6760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:31:02.312883 6775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.313064 6760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:02.313180 6760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "0079b7e1912f4abab6b1e62068197aaf"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "ed263a43b3cdd3ffd682afb01de7b884"
server_key_iv: "2301e032be54df862b986094133c476f"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.313273 6760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:02.330493 6760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:02.332504 6760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:02.332669 6760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:02.332914 6760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:02.333228 6760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:02.333281 6760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.333328 6760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:02.333361 6760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.339331 6760 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:45035
I20260619 20:31:02.339389 6889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:45035 every 8 connection(s)
I20260619 20:31:02.339807 6760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:31:02 dist-test-slave-f9rk krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901062, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:02.349282 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 6760
I20260619 20:31:02.349411 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:31:02.349459 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.340569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:53513 (local address 127.21.151.62:43227)
0619 20:31:02.340635 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.340637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.340833 (+ 196us) server_negotiation.cc:407] Connection header received
0619 20:31:02.341313 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.341314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.341323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.341381 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:02.342339 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.342834 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.344174 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.344510 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.344596 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.346894 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:02.346895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:02.346896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:02.346902 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:02.348202 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:02.348576 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:02.348577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:02.348577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:02.348626 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:02.348854 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:02.348854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:02.348855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:02.348955 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:02.349005 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.349274 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.349364 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 20:31:02.349640 22108 external_mini_cluster.cc:1428] Setting key c70c106999e7f9d5fca8859a37cd92ae
I20260619 20:31:02.349659 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.340554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:43227 (local address 127.21.151.1:53513)
0619 20:31:02.340701 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:02.340745 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:02.341229 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:02.341442 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:02.341446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:02.342051 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:02.342250 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.342256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.342929 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.342930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.344078 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.344081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.344679 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.344680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.344855 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.345429 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:02.345439 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:02.346796 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:02.348274 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:02.348281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:02.348282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:02.348511 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:02.348683 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:02.348683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:02.348684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:02.348782 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:02.349010 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:02.349015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:02.349131 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 20:31:02.349245 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 20:31:02.349401 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260619 20:31:02.350216 6890 heartbeater.cc:347] Connected to a master server at 127.21.151.62:43227
I20260619 20:31:02.350335 6890 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:02.350543 6890 heartbeater.cc:511] Master 127.21.151.62:43227 requested a full tablet report, sending...
I20260619 20:31:02.351073 6698 ts_manager.cc:194] Registered new tserver with Master: 0079b7e1912f4abab6b1e62068197aaf (127.21.151.1:45035)
I20260619 20:31:02.351751 6698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:53513
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:31:02.372695 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:43227
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:02.454664 6895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:02.454926 6895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:02.454967 6895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:02.456220 6895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:02.456298 6895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:02.456343 6895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:31:02.457878 6895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:43227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:02.458141 6895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:02.458367 6895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:02.463341 6900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:02.463337 6903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:02.463335 6901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:02.463668 6895 server_base.cc:1034] running on GCE node
I20260619 20:31:02.463878 6895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:02.464077 6895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:02.465222 6895 hybrid_clock.cc:648] HybridClock initialized: now 1781901062465210 us; error 26 us; skew 500 ppm
Jun 19 20:31:02 dist-test-slave-f9rk krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901062, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:02.467936 6895 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:31:02.468330 6895 webserver.cc:460] Webserver started at http://127.21.151.2:34623/ using document root <none> and password file <none>
I20260619 20:31:02.468530 6895 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:02.468570 6895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:02.468660 6895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:02.469646 6895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "8796a89091364e05b00790dca4a6a276"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "c3b56ab46ca46674f62e04c38302b553"
server_key_iv: "8144ffa904a249ea928746f109da2d74"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.469944 6895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "8796a89091364e05b00790dca4a6a276"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "c3b56ab46ca46674f62e04c38302b553"
server_key_iv: "8144ffa904a249ea928746f109da2d74"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.471148 6895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:02.471830 6910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.472000 6895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:02.472095 6895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "8796a89091364e05b00790dca4a6a276"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "c3b56ab46ca46674f62e04c38302b553"
server_key_iv: "8144ffa904a249ea928746f109da2d74"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.472179 6895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:02.492905 6895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:02.494813 6895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:02.494997 6895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:02.495220 6895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:02.495543 6895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:02.495595 6895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.495657 6895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:02.495702 6895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.501434 6895 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:46225
I20260619 20:31:02.501477 7023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:46225 every 8 connection(s)
I20260619 20:31:02.501784 6895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
Jun 19 20:31:02 dist-test-slave-f9rk krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901062, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:02.508080 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 6895
I20260619 20:31:02.508232 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:31:02.508446 22108 external_mini_cluster.cc:1428] Setting key e99f409e468e4c5edc042ee9a9289f79
I20260619 20:31:02.511384 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.502512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:49057 (local address 127.21.151.62:43227)
0619 20:31:02.502594 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.502597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.502770 (+ 173us) server_negotiation.cc:407] Connection header received
0619 20:31:02.503225 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.503226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.503235 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.503288 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:02.504388 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.505199 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.506155 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.506386 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.506455 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.508604 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:02.508605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:02.508606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:02.508614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:02.510012 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:02.510416 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:02.510417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:02.510417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:02.510479 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:02.510734 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:02.510735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:02.510735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:02.510848 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:02.510932 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.511217 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.511298 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 20:31:02.511575 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.502443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:43227 (local address 127.21.151.2:49057)
0619 20:31:02.502648 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:02.502686 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:02.503147 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:02.503364 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:02.503368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:02.503997 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:02.504276 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.504283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.505281 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.505282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.506078 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.506080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.506458 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.506458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.506557 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.506999 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:02.507007 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:02.508519 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:02.510099 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:02.510106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:02.510108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:02.510325 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:02.510549 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:02.510550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:02.510550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:02.510666 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:02.510917 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:02.510925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:02.511067 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 20:31:02.511182 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 20:31:02.511335 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":124,"threads_started":1}
I20260619 20:31:02.512005 7024 heartbeater.cc:347] Connected to a master server at 127.21.151.62:43227
I20260619 20:31:02.512116 7024 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:02.512292 7024 heartbeater.cc:511] Master 127.21.151.62:43227 requested a full tablet report, sending...
I20260619 20:31:02.512658 6698 ts_manager.cc:194] Registered new tserver with Master: 8796a89091364e05b00790dca4a6a276 (127.21.151.2:46225)
I20260619 20:31:02.513087 6698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:49057
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:31:02.532696 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:43227
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:02.612360 7029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:02.612589 7029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:02.612627 7029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:02.613763 7029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:02.613835 7029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:02.613875 7029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:31:02.615345 7029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:43227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:02.615612 7029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:02.615859 7029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:02.620797 7034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:02.620787 7035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:02.620842 7037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:02.620935 7029 server_base.cc:1034] running on GCE node
I20260619 20:31:02.621440 7029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:02.621681 7029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:02.622860 7029 hybrid_clock.cc:648] HybridClock initialized: now 1781901062622853 us; error 37 us; skew 500 ppm
Jun 19 20:31:02 dist-test-slave-f9rk krb5kdc[6653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901062, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:02.625329 7029 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:31:02.625758 7029 webserver.cc:460] Webserver started at http://127.21.151.3:43453/ using document root <none> and password file <none>
I20260619 20:31:02.626019 7029 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:02.626089 7029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:02.626235 7029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:02.627312 7029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "45d6238f589c4206bcf6de1d86084f9d"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "62482aecc53c4fbc0ec9c41ce9987fe9"
server_key_iv: "8c06f3898b8305d6b06c4652766bef56"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.627745 7029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "45d6238f589c4206bcf6de1d86084f9d"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "62482aecc53c4fbc0ec9c41ce9987fe9"
server_key_iv: "8c06f3898b8305d6b06c4652766bef56"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.629112 7029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:31:02.629838 7044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.630015 7029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:02.630115 7029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "45d6238f589c4206bcf6de1d86084f9d"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "62482aecc53c4fbc0ec9c41ce9987fe9"
server_key_iv: "8c06f3898b8305d6b06c4652766bef56"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.630196 7029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:02.665796 7029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:02.668146 7029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:02.668329 7029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:02.668623 7029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:02.668977 7029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:02.669010 7029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.669032 7029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:02.669045 7029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.674970 7029 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:46471
I20260619 20:31:02.675046 7157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:46471 every 8 connection(s)
I20260619 20:31:02.675371 7029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:31:02.677433 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 7029
I20260619 20:31:02.677543 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:31:02.677688 22108 external_mini_cluster.cc:1428] Setting key 486200c6ef16659624e3ee36c3b255c3
Jun 19 20:31:02 dist-test-slave-f9rk krb5kdc[6653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901062, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:02.684957 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.676243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:33207 (local address 127.21.151.62:43227)
0619 20:31:02.676312 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.676313 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.676515 (+ 202us) server_negotiation.cc:407] Connection header received
0619 20:31:02.677120 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.677121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.677133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.677191 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:02.678328 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.678830 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.679910 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.680113 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.680198 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.682328 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:02.682329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:02.682330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:02.682338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:02.683532 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:02.683980 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:02.683981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:02.683982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:02.684051 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:02.684306 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:02.684307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:02.684307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:02.684428 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:02.684505 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.684780 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.684859 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:31:02.685170 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.676225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:43227 (local address 127.21.151.3:33207)
0619 20:31:02.676392 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:02.676434 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:02.677002 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:02.677286 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:02.677292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:02.678023 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:02.678221 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.678226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.678935 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.678936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.679827 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.679829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.680207 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.680208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.680339 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.680870 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:02.680879 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:02.682226 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:02.683651 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:02.683659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:02.683661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:02.683888 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:02.684127 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:02.684127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:02.684128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:02.684239 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:02.684496 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:02.684502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:02.684641 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 20:31:02.684759 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 20:31:02.684936 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260619 20:31:02.685654 7158 heartbeater.cc:347] Connected to a master server at 127.21.151.62:43227
I20260619 20:31:02.685763 7158 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:02.685945 7158 heartbeater.cc:511] Master 127.21.151.62:43227 requested a full tablet report, sending...
I20260619 20:31:02.686342 6698 ts_manager.cc:194] Registered new tserver with Master: 45d6238f589c4206bcf6de1d86084f9d (127.21.151.3:46471)
I20260619 20:31:02.686721 6698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:33207
I20260619 20:31:02.689321 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:31:02.696521 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.690653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.21.151.62:43227)
0619 20:31:02.690731 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.690733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.690832 (+ 99us) server_negotiation.cc:407] Connection header received
0619 20:31:02.690907 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.690908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.690918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.690967 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:02.691503 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.692039 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.692757 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.692961 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.693038 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.694133 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:02.694135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:02.694135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:02.694142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:02.695345 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:02.695718 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:02.695719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:02.695719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:02.695777 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:02.695986 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:02.695987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:02.695987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:02.696144 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:02.696244 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.696346 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.696427 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 20:31:02.698140 6698 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:31:02.699718 6698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:31:02.706089 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.701996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:45035 (local address 127.0.0.1:33202)
0619 20:31:02.702392 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:02.702402 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:02.702451 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:02.702674 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:02.702676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:02.702678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:02.702743 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.702745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.704052 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.704053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.705118 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.705120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.705790 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.705790 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.705870 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.705883 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:31:02.705958 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:31:02.706013 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":7,"thread_start_us":82,"threads_started":1}
I20260619 20:31:02.706099 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.702104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.21.151.1:45035)
0619 20:31:02.702296 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.702299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.702449 (+ 150us) server_negotiation.cc:407] Connection header received
0619 20:31:02.702555 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.702558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.702575 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.702651 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:02.702823 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.703950 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.705204 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.705702 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.705791 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.705852 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.705973 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.706010 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
I20260619 20:31:02.706411 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.702403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.21.151.2:46225)
0619 20:31:02.702586 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.702591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.702600 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:31:02.702808 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.702810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.702827 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.702898 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:02.703117 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.704140 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.705247 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.705918 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.705973 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.706023 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.706273 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.706321 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
I20260619 20:31:02.706542 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.702076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.21.151.3:46471)
0619 20:31:02.702247 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.702252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.702386 (+ 134us) server_negotiation.cc:407] Connection header received
0619 20:31:02.702507 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.702509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.702527 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.702616 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:02.702843 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.703699 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.705158 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.705900 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.705972 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.706024 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.706415 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.706464 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260619 20:31:02.706331 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.701987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:46471 (local address 127.0.0.1:55442)
0619 20:31:02.702266 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:02.702309 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:02.702390 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:02.702699 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:02.702700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:02.702702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:02.702761 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.702763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.703802 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.703803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.705068 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.705070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.706062 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.706062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.706134 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.706137 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:02.706180 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 20:31:02.706228 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":9,"thread_start_us":54,"threads_started":1}
I20260619 20:31:02.706336 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.702183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:46225 (local address 127.0.0.1:58002)
0619 20:31:02.702509 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:02.702517 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:02.702552 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:02.702988 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:02.702988 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:02.702990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:02.703032 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.703033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.704240 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.704241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.705156 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.705158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.706060 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.706061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.706126 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.706130 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:31:02.706180 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:31:02.706228 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260619 20:31:02.707187 6823 tablet_service.cc:1467] Processing CreateTablet for tablet f03fbafe7c3d4fde85c581f1f95d42e4 (DEFAULT_TABLE table=test-table [id=09ac62bd2dee4b7883eb4823d2913b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:02.707232 7092 tablet_service.cc:1467] Processing CreateTablet for tablet f03fbafe7c3d4fde85c581f1f95d42e4 (DEFAULT_TABLE table=test-table [id=09ac62bd2dee4b7883eb4823d2913b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:02.707547 6823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03fbafe7c3d4fde85c581f1f95d42e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:02.707547 7092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03fbafe7c3d4fde85c581f1f95d42e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:02.710881 7175 tablet_bootstrap.cc:492] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: Bootstrap starting.
I20260619 20:31:02.711077 7176 tablet_bootstrap.cc:492] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf: Bootstrap starting.
I20260619 20:31:02.711439 6958 tablet_service.cc:1467] Processing CreateTablet for tablet f03fbafe7c3d4fde85c581f1f95d42e4 (DEFAULT_TABLE table=test-table [id=09ac62bd2dee4b7883eb4823d2913b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:02.711743 7175 tablet_bootstrap.cc:654] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:02.711743 7176 tablet_bootstrap.cc:654] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:02.711803 6958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03fbafe7c3d4fde85c581f1f95d42e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:02.712388 7175 log.cc:824] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:02.712388 7176 log.cc:824] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:02.713088 7176 tablet_bootstrap.cc:492] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf: No bootstrap required, opened a new log
I20260619 20:31:02.713131 7175 tablet_bootstrap.cc:492] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: No bootstrap required, opened a new log
I20260619 20:31:02.713171 7176 ts_tablet_manager.cc:1397] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:31:02.713189 7175 ts_tablet_manager.cc:1397] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 20:31:02.714712 7179 tablet_bootstrap.cc:492] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: Bootstrap starting.
I20260619 20:31:02.714814 7176 raft_consensus.cc:348] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0079b7e1912f4abab6b1e62068197aaf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45035 } } peers { permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 } } peers { permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 } }
I20260619 20:31:02.715009 7176 raft_consensus.cc:374] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:02.715055 7176 raft_consensus.cc:729] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0079b7e1912f4abab6b1e62068197aaf, State: Initialized, Role: FOLLOWER
I20260619 20:31:02.715227 7176 consensus_queue.cc:260] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0079b7e1912f4abab6b1e62068197aaf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45035 } } peers { permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 } } peers { permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 } }
I20260619 20:31:02.715262 7175 raft_consensus.cc:348] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0079b7e1912f4abab6b1e62068197aaf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45035 } } peers { permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 } } peers { permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 } }
I20260619 20:31:02.715452 7179 tablet_bootstrap.cc:654] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:02.715519 6890 heartbeater.cc:503] Master 127.21.151.62:43227 was elected leader, sending a full tablet report...
I20260619 20:31:02.715540 7176 ts_tablet_manager.cc:1428] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:31:02.715562 7175 raft_consensus.cc:374] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:02.715607 7175 raft_consensus.cc:729] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d6238f589c4206bcf6de1d86084f9d, State: Initialized, Role: FOLLOWER
I20260619 20:31:02.715736 7175 consensus_queue.cc:260] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0079b7e1912f4abab6b1e62068197aaf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45035 } } peers { permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 } } peers { permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 } }
I20260619 20:31:02.715996 7175 ts_tablet_manager.cc:1428] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:31:02.716078 7179 log.cc:824] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:02.716145 7158 heartbeater.cc:503] Master 127.21.151.62:43227 was elected leader, sending a full tablet report...
I20260619 20:31:02.717000 7179 tablet_bootstrap.cc:492] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: No bootstrap required, opened a new log
I20260619 20:31:02.717082 7179 ts_tablet_manager.cc:1397] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:31:02.718508 7179 raft_consensus.cc:348] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0079b7e1912f4abab6b1e62068197aaf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45035 } } peers { permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 } } peers { permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 } }
I20260619 20:31:02.718657 7179 raft_consensus.cc:374] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:02.718708 7179 raft_consensus.cc:729] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8796a89091364e05b00790dca4a6a276, State: Initialized, Role: FOLLOWER
I20260619 20:31:02.718812 7179 consensus_queue.cc:260] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0079b7e1912f4abab6b1e62068197aaf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45035 } } peers { permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 } } peers { permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 } }
I20260619 20:31:02.719098 7179 ts_tablet_manager.cc:1428] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:31:02.719106 7024 heartbeater.cc:503] Master 127.21.151.62:43227 was elected leader, sending a full tablet report...
I20260619 20:31:02.736680 7180 raft_consensus.cc:482] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:31:02.736835 7180 raft_consensus.cc:504] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0079b7e1912f4abab6b1e62068197aaf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45035 } } peers { permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 } } peers { permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 } }
I20260619 20:31:02.737219 7180 leader_election.cc:288] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45d6238f589c4206bcf6de1d86084f9d (127.21.151.3:46471), 8796a89091364e05b00790dca4a6a276 (127.21.151.2:46225)
I20260619 20:31:02.740192 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.737495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:47845 (local address 127.21.151.2:46225)
0619 20:31:02.737561 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.737564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.737573 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:31:02.737602 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.737603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.737617 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.737685 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:02.737858 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.738330 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.739385 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.739824 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.739915 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.739923 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.740083 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.740118 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:31:02.740195 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.737555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:50131 (local address 127.21.151.3:46471)
0619 20:31:02.737603 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.737606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.737699 (+ 93us) server_negotiation.cc:407] Connection header received
0619 20:31:02.737762 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.737763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.737772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.737843 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:02.738001 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.738455 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.739411 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.739844 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.739933 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.739942 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.740092 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.740127 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:31:02.740221 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.737378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:46225 (local address 127.21.151.1:47845)
0619 20:31:02.737442 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:02.737455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:02.737503 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:02.737703 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:02.737703 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:02.737705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:02.737781 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.737783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.738412 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.738413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.739274 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.739276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.739910 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.739911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.740001 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.740004 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:02.740066 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 20:31:02.740109 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":61}
I20260619 20:31:02.740233 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.737385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:46471 (local address 127.21.151.1:50131)
0619 20:31:02.737611 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:02.737625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:02.737704 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:02.737845 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:02.737846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:02.737848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:02.737928 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.737930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.738537 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.738538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.739317 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:02.739319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.739924 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.739925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:02.740007 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.740010 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:02.740084 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 20:31:02.740124 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":49,"thread_start_us":105,"threads_started":1}
I20260619 20:31:02.740667 6978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03fbafe7c3d4fde85c581f1f95d42e4" candidate_uuid: "0079b7e1912f4abab6b1e62068197aaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8796a89091364e05b00790dca4a6a276" is_pre_election: true
I20260619 20:31:02.740667 7112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03fbafe7c3d4fde85c581f1f95d42e4" candidate_uuid: "0079b7e1912f4abab6b1e62068197aaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d6238f589c4206bcf6de1d86084f9d" is_pre_election: true
I20260619 20:31:02.740820 7112 raft_consensus.cc:2454] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0079b7e1912f4abab6b1e62068197aaf in term 0.
I20260619 20:31:02.740820 6978 raft_consensus.cc:2454] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0079b7e1912f4abab6b1e62068197aaf in term 0.
I20260619 20:31:02.741074 6777 leader_election.cc:302] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0079b7e1912f4abab6b1e62068197aaf, 8796a89091364e05b00790dca4a6a276; no voters:
I20260619 20:31:02.741259 7180 raft_consensus.cc:2781] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:31:02.741319 7180 raft_consensus.cc:482] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:31:02.741344 7180 raft_consensus.cc:3037] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:02.742076 7180 raft_consensus.cc:504] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0079b7e1912f4abab6b1e62068197aaf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45035 } } peers { permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 } } peers { permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 } }
I20260619 20:31:02.742233 7180 leader_election.cc:288] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [CANDIDATE]: Term 1 election: Requested vote from peers 45d6238f589c4206bcf6de1d86084f9d (127.21.151.3:46471), 8796a89091364e05b00790dca4a6a276 (127.21.151.2:46225)
I20260619 20:31:02.742383 6978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03fbafe7c3d4fde85c581f1f95d42e4" candidate_uuid: "0079b7e1912f4abab6b1e62068197aaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8796a89091364e05b00790dca4a6a276"
I20260619 20:31:02.742452 7112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03fbafe7c3d4fde85c581f1f95d42e4" candidate_uuid: "0079b7e1912f4abab6b1e62068197aaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d6238f589c4206bcf6de1d86084f9d"
I20260619 20:31:02.742491 6978 raft_consensus.cc:3037] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:02.742518 7112 raft_consensus.cc:3037] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:02.743408 6978 raft_consensus.cc:2454] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0079b7e1912f4abab6b1e62068197aaf in term 1.
I20260619 20:31:02.743408 7112 raft_consensus.cc:2454] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0079b7e1912f4abab6b1e62068197aaf in term 1.
I20260619 20:31:02.743577 6777 leader_election.cc:302] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0079b7e1912f4abab6b1e62068197aaf, 8796a89091364e05b00790dca4a6a276; no voters:
I20260619 20:31:02.743726 7180 raft_consensus.cc:2781] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:02.743911 7180 raft_consensus.cc:686] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [term 1 LEADER]: Becoming Leader. State: Replica: 0079b7e1912f4abab6b1e62068197aaf, State: Running, Role: LEADER
I20260619 20:31:02.744047 7180 consensus_queue.cc:237] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0079b7e1912f4abab6b1e62068197aaf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45035 } } peers { permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 } } peers { permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 } }
I20260619 20:31:02.744877 6697 catalog_manager.cc:5495] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0079b7e1912f4abab6b1e62068197aaf (127.21.151.1). New cstate: current_term: 1 leader_uuid: "0079b7e1912f4abab6b1e62068197aaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0079b7e1912f4abab6b1e62068197aaf" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 } health_report { overall_health: UNKNOWN } } }
W20260619 20:31:02.752357 7025 tablet.cc:2369] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:31:02.755527 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.753048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.21.151.1:45035)
0619 20:31:02.753097 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.753099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.753118 (+ 19us) server_negotiation.cc:407] Connection header received
0619 20:31:02.753158 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.753158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.753182 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.753262 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:02.753370 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.753926 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.754710 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.754908 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.754985 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.755157 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:02.755264 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:02.755362 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.755387 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.755407 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:31:02.757316 7112 raft_consensus.cc:1261] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d [term 1 FOLLOWER]: Refusing update from remote peer 0079b7e1912f4abab6b1e62068197aaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:02.757318 6978 raft_consensus.cc:1261] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276 [term 1 FOLLOWER]: Refusing update from remote peer 0079b7e1912f4abab6b1e62068197aaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:02.757669 7185 consensus_queue.cc:1035] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d6238f589c4206bcf6de1d86084f9d" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 46471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:02.757802 7180 consensus_queue.cc:1035] T f03fbafe7c3d4fde85c581f1f95d42e4 P 0079b7e1912f4abab6b1e62068197aaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8796a89091364e05b00790dca4a6a276" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 46225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:02.759987 7188 mvcc.cc:205] Tried to move back new op lower bound from 7298666753051254784 to 7298666753000243200. Current Snapshot: MvccSnapshot[applied={T|T < 7298666753051254784}]
I20260619 20:31:02.760464 7190 mvcc.cc:205] Tried to move back new op lower bound from 7298666753051254784 to 7298666753000243200. Current Snapshot: MvccSnapshot[applied={T|T < 7298666753051254784}]
I20260619 20:31:02.762759 6698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:02.762862 6698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:02.763671 6698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3245962e1f341ef858609aef6a51713: Sending DeleteTablet for 3 replicas of tablet f03fbafe7c3d4fde85c581f1f95d42e4
I20260619 20:31:02.764034 6958 tablet_service.cc:1514] Processing DeleteTablet for tablet f03fbafe7c3d4fde85c581f1f95d42e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:02 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:58002
I20260619 20:31:02.764036 7092 tablet_service.cc:1514] Processing DeleteTablet for tablet f03fbafe7c3d4fde85c581f1f95d42e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:02 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:55442
I20260619 20:31:02.764089 6823 tablet_service.cc:1514] Processing DeleteTablet for tablet f03fbafe7c3d4fde85c581f1f95d42e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:02 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:33202
I20260619 20:31:02.764251 7198 tablet_replica.cc:330] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: stopping tablet replica
I20260619 20:31:02.764345 7198 raft_consensus.cc:2229] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:31:02.764410 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 6760
I20260619 20:31:02.764454 7198 raft_consensus.cc:2258] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:02.765401 7198 ts_tablet_manager.cc:1905] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:02.766955 7198 ts_tablet_manager.cc:1918] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:02.767059 7198 log.cc:1196] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/f03fbafe7c3d4fde85c581f1f95d42e4
I20260619 20:31:02.767304 7198 ts_tablet_manager.cc:1939] T f03fbafe7c3d4fde85c581f1f95d42e4 P 8796a89091364e05b00790dca4a6a276: Deleting consensus metadata
I20260619 20:31:02.767689 7199 tablet_replica.cc:330] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: stopping tablet replica
I20260619 20:31:02.767730 6683 catalog_manager.cc:4841] TS 8796a89091364e05b00790dca4a6a276 (127.21.151.2:46225): tablet f03fbafe7c3d4fde85c581f1f95d42e4 (table test-table [id=09ac62bd2dee4b7883eb4823d2913b33]) successfully deleted
I20260619 20:31:02.767810 7199 raft_consensus.cc:2229] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:31:02.767915 7199 raft_consensus.cc:2258] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:02.768296 7199 ts_tablet_manager.cc:1905] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:02.769716 7199 ts_tablet_manager.cc:1918] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:02.769775 7199 log.cc:1196] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/f03fbafe7c3d4fde85c581f1f95d42e4
I20260619 20:31:02.770028 7199 ts_tablet_manager.cc:1939] T f03fbafe7c3d4fde85c581f1f95d42e4 P 45d6238f589c4206bcf6de1d86084f9d: Deleting consensus metadata
I20260619 20:31:02.770435 6684 catalog_manager.cc:4841] TS 45d6238f589c4206bcf6de1d86084f9d (127.21.151.3:46471): tablet f03fbafe7c3d4fde85c581f1f95d42e4 (table test-table [id=09ac62bd2dee4b7883eb4823d2913b33]) successfully deleted
W20260619 20:31:02.771327 6683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:45035 (error 108)
I20260619 20:31:02.771524 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 6895
I20260619 20:31:02.772601 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.772467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:45035 (local address 127.0.0.1:33220)
0619 20:31:02.772526 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:02.772565 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:45035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 20:31:02.772758 6683 catalog_manager.cc:4584] TS 0079b7e1912f4abab6b1e62068197aaf (127.21.151.1:45035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f03fbafe7c3d4fde85c581f1f95d42e4: Network error: Client connection negotiation failed: client connection to 127.21.151.1:45035: connect: Connection refused (error 111)
I20260619 20:31:02.777681 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 7029
I20260619 20:31:02.782321 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 6666
2026-06-19T20:31:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1885 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T20:31:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:02Z Disabled control of system clock
I20260619 20:31:02.809785 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:33501
--webserver_interface=127.21.151.62
--webserver_port=0
--builtin_ntp_servers=127.21.151.20:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:33501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:31:02.887917 7205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:02.888113 7205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:02.888132 7205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:02.889147 7205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:02.889178 7205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:02.889191 7205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:02.889204 7205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:02.889214 7205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:02.890481 7205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:33501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:33501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:02.890661 7205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:02.890887 7205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:02.895289 7213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:02.895360 7210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:02.895440 7205 server_base.cc:1034] running on GCE node
W20260619 20:31:02.895313 7211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:02.895773 7205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:02.896047 7205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:02.897193 7205 hybrid_clock.cc:648] HybridClock initialized: now 1781901062897187 us; error 32 us; skew 500 ppm
I20260619 20:31:02.898332 7205 webserver.cc:460] Webserver started at http://127.21.151.62:43703/ using document root <none> and password file <none>
I20260619 20:31:02.898573 7205 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:02.898618 7205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:02.898774 7205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:02.899849 7205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "60493796799d49bdb8d89d7efbcdd148"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "6f69dd7b697b29bd01428385e8522442"
server_key_iv: "b6975fda2b3e126713c011d794548254"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.900193 7205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "60493796799d49bdb8d89d7efbcdd148"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "6f69dd7b697b29bd01428385e8522442"
server_key_iv: "b6975fda2b3e126713c011d794548254"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.901433 7205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:02.902107 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:02.902273 7205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:02.902333 7205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "60493796799d49bdb8d89d7efbcdd148"
format_stamp: "Formatted at 2026-06-19 20:31:02 on dist-test-slave-f9rk"
server_key: "6f69dd7b697b29bd01428385e8522442"
server_key_iv: "b6975fda2b3e126713c011d794548254"
server_key_version: "encryptionkey@0"
I20260619 20:31:02.902419 7205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:02.914125 7205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:02.914409 7205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:02.914551 7205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:02.918428 7205 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:33501
I20260619 20:31:02.918488 7271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:33501 every 8 connection(s)
I20260619 20:31:02.918800 7205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:02.919801 7272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:02.922036 7272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148: Bootstrap starting.
I20260619 20:31:02.922602 7272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:02.922906 7272 log.cc:824] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:02.923555 7272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148: No bootstrap required, opened a new log
I20260619 20:31:02.923877 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 7205
I20260619 20:31:02.923993 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:02.924189 22108 external_mini_cluster.cc:1428] Setting key 4543f751435103972b68a9afc2780e68
I20260619 20:31:02.925851 7272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60493796799d49bdb8d89d7efbcdd148" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33501 } }
I20260619 20:31:02.926033 7272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:02.926072 7272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60493796799d49bdb8d89d7efbcdd148, State: Initialized, Role: FOLLOWER
I20260619 20:31:02.926215 7272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60493796799d49bdb8d89d7efbcdd148" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33501 } }
I20260619 20:31:02.926303 7272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:02.926334 7272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:02.926376 7272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:02.927090 7272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60493796799d49bdb8d89d7efbcdd148" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33501 } }
I20260619 20:31:02.927196 7272 leader_election.cc:302] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60493796799d49bdb8d89d7efbcdd148; no voters:
I20260619 20:31:02.927347 7272 leader_election.cc:288] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:02.927409 7277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:02.927552 7277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [term 1 LEADER]: Becoming Leader. State: Replica: 60493796799d49bdb8d89d7efbcdd148, State: Running, Role: LEADER
I20260619 20:31:02.927577 7272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:02.927656 7277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60493796799d49bdb8d89d7efbcdd148" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33501 } }
I20260619 20:31:02.927963 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60493796799d49bdb8d89d7efbcdd148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60493796799d49bdb8d89d7efbcdd148" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33501 } } }
I20260619 20:31:02.927948 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60493796799d49bdb8d89d7efbcdd148. Latest consensus state: current_term: 1 leader_uuid: "60493796799d49bdb8d89d7efbcdd148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60493796799d49bdb8d89d7efbcdd148" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33501 } } }
I20260619 20:31:02.928042 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:02.928514 7284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:02.928050 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:02.929003 7284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:02.931463 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:02.924719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.21.151.62:33501)
0619 20:31:02.924944 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:02.924955 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:02.924987 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:31:02.925804 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:02.925813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:02.925894 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:02.926048 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:02.926261 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.928006 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.929083 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:02.929834 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:02.929952 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:02.930760 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:02.930764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:02.930765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:02.930971 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:02.931011 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:02.931025 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:02.931087 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:02.931193 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 20:31:02.931272 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260619 20:31:02.931766 7284 catalog_manager.cc:1347] Generated new cluster ID: 6b0cab2a26534a50920c32254f7fdc0c
I20260619 20:31:02.931818 7284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:02.959192 7284 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:02.959890 7284 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:02.963840 7284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148: Generated new TSK 0
I20260619 20:31:02.964036 7284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:02.968501 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33501
--builtin_ntp_servers=127.21.151.20:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:31:03.051026 7296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:03.051226 7296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:03.051247 7296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:03.052450 7296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:03.052489 7296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:03.052515 7296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:31:03.053871 7296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:03.054124 7296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:03.054421 7296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:03.059901 7301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:03.059926 7302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:03.060184 7304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:03.060220 7296 server_base.cc:1034] running on GCE node
I20260619 20:31:03.060390 7296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:03.060600 7296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:03.061740 7296 hybrid_clock.cc:648] HybridClock initialized: now 1781901063061729 us; error 28 us; skew 500 ppm
I20260619 20:31:03.062927 7296 webserver.cc:460] Webserver started at http://127.21.151.1:37229/ using document root <none> and password file <none>
I20260619 20:31:03.063113 7296 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:03.063153 7296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:03.063241 7296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:03.064308 7296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "7e88ab4d14034a68b6092151ec0bbb11"
format_stamp: "Formatted at 2026-06-19 20:31:03 on dist-test-slave-f9rk"
server_key: "33969e7a1f4e7d4ad8e660dc39d8b48e"
server_key_iv: "8df1be7fba6c92e01be6fa8e3c8df314"
server_key_version: "encryptionkey@0"
I20260619 20:31:03.064615 7296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "7e88ab4d14034a68b6092151ec0bbb11"
format_stamp: "Formatted at 2026-06-19 20:31:03 on dist-test-slave-f9rk"
server_key: "33969e7a1f4e7d4ad8e660dc39d8b48e"
server_key_iv: "8df1be7fba6c92e01be6fa8e3c8df314"
server_key_version: "encryptionkey@0"
I20260619 20:31:03.065989 7296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:03.075809 7310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:03.076304 7296 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260619 20:31:03.076416 7296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "7e88ab4d14034a68b6092151ec0bbb11"
format_stamp: "Formatted at 2026-06-19 20:31:03 on dist-test-slave-f9rk"
server_key: "33969e7a1f4e7d4ad8e660dc39d8b48e"
server_key_iv: "8df1be7fba6c92e01be6fa8e3c8df314"
server_key_version: "encryptionkey@0"
I20260619 20:31:03.076550 7296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:03.087913 7296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:03.088209 7296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:03.088328 7296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:03.088510 7296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:03.088773 7296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:03.088802 7296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:03.088824 7296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:03.088836 7296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:03.094568 7296 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:46005
I20260619 20:31:03.094609 7423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:46005 every 8 connection(s)
I20260619 20:31:03.095008 7296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:31:03.096067 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 7296
I20260619 20:31:03.096190 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:31:03.096392 22108 external_mini_cluster.cc:1428] Setting key 19bcb45035645760f2cc4af613f29ea4
I20260619 20:31:03.097383 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33501
--builtin_ntp_servers=127.21.151.20:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 20:31:03.101204 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.095711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:42609 (local address 127.21.151.62:33501)
0619 20:31:03.095778 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:03.095780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:03.096023 (+ 243us) server_negotiation.cc:407] Connection header received
0619 20:31:03.096745 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:03.096746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:03.096758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:03.096815 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:03.097677 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.098167 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.099727 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.099945 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.100025 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.100932 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:03.100934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:03.100934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:03.100939 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:03.100954 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:03.100957 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:03.101008 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:03.101097 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 20:31:03.101126 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:31:03.101456 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.095726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33501 (local address 127.21.151.1:42609)
0619 20:31:03.095903 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:03.095938 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:03.096632 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:03.096886 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:03.096890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:03.096899 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:03.097569 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.097576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.098259 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.098260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.099652 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.099655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.100034 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.100035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.100179 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.100766 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:31:03.100766 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:03.100796 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:31:03.100819 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:03.101019 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:03.101024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:03.101027 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:03.101099 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 20:31:03.101202 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260619 20:31:03.101933 7424 heartbeater.cc:347] Connected to a master server at 127.21.151.62:33501
I20260619 20:31:03.102030 7424 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:03.102276 7424 heartbeater.cc:511] Master 127.21.151.62:33501 requested a full tablet report, sending...
I20260619 20:31:03.102818 7236 ts_manager.cc:194] Registered new tserver with Master: 7e88ab4d14034a68b6092151ec0bbb11 (127.21.151.1:46005)
I20260619 20:31:03.103484 7236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.1:42609
W20260619 20:31:03.178071 7427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:03.178300 7427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:03.178345 7427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:03.179489 7427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:03.179571 7427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:03.179656 7427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:31:03.181165 7427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:03.181417 7427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:03.181744 7427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:03.187341 7433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:03.187347 7432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:03.187569 7435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:03.187858 7427 server_base.cc:1034] running on GCE node
I20260619 20:31:03.188067 7427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:03.188326 7427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:03.189478 7427 hybrid_clock.cc:648] HybridClock initialized: now 1781901063189463 us; error 37 us; skew 500 ppm
I20260619 20:31:03.190656 7427 webserver.cc:460] Webserver started at http://127.21.151.2:36719/ using document root <none> and password file <none>
I20260619 20:31:03.190932 7427 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:03.191001 7427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:03.191149 7427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:03.192298 7427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "05ee3a84af5e4366b3154c6078ec4cec"
format_stamp: "Formatted at 2026-06-19 20:31:03 on dist-test-slave-f9rk"
server_key: "51fdcac02380f34b6d3dae2ea0df7b9d"
server_key_iv: "8f4f4fd831da2cacb0ba7f7146af1bfa"
server_key_version: "encryptionkey@0"
I20260619 20:31:03.192652 7427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "05ee3a84af5e4366b3154c6078ec4cec"
format_stamp: "Formatted at 2026-06-19 20:31:03 on dist-test-slave-f9rk"
server_key: "51fdcac02380f34b6d3dae2ea0df7b9d"
server_key_iv: "8f4f4fd831da2cacb0ba7f7146af1bfa"
server_key_version: "encryptionkey@0"
I20260619 20:31:03.193956 7427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:03.194705 7441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:03.194883 7427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:03.194988 7427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "05ee3a84af5e4366b3154c6078ec4cec"
format_stamp: "Formatted at 2026-06-19 20:31:03 on dist-test-slave-f9rk"
server_key: "51fdcac02380f34b6d3dae2ea0df7b9d"
server_key_iv: "8f4f4fd831da2cacb0ba7f7146af1bfa"
server_key_version: "encryptionkey@0"
I20260619 20:31:03.195070 7427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:03.217576 7427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:03.217902 7427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:03.218057 7427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:03.218289 7427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:03.218617 7427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:03.218669 7427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:03.218709 7427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:03.218735 7427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:03.224464 7427 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:34049
I20260619 20:31:03.224526 7554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:34049 every 8 connection(s)
I20260619 20:31:03.224895 7427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:31:03.230554 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.225672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:44831 (local address 127.21.151.62:33501)
0619 20:31:03.225728 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:03.225730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:03.225974 (+ 244us) server_negotiation.cc:407] Connection header received
0619 20:31:03.226523 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:03.226525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:03.226538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:03.226592 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:03.227350 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.228008 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.229121 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.229306 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.229379 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.230252 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:03.230254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:03.230254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:03.230258 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:03.230280 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:03.230284 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:03.230336 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:03.230451 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 20:31:03.230476 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:31:03.230772 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.225662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33501 (local address 127.21.151.2:44831)
0619 20:31:03.225848 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:03.225881 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:03.226410 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:03.226682 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:03.226685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:03.226693 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:03.227254 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.227259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.228106 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.228108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.229027 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.229030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.229389 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.229390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.229519 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.230102 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:31:03.230102 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:03.230130 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:31:03.230153 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:03.230353 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:03.230359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:03.230362 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:03.230447 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:31:03.230547 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260619 20:31:03.231268 7555 heartbeater.cc:347] Connected to a master server at 127.21.151.62:33501
I20260619 20:31:03.231354 7555 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:03.231556 7555 heartbeater.cc:511] Master 127.21.151.62:33501 requested a full tablet report, sending...
I20260619 20:31:03.231961 7236 ts_manager.cc:194] Registered new tserver with Master: 05ee3a84af5e4366b3154c6078ec4cec (127.21.151.2:34049)
I20260619 20:31:03.232287 7236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.2:44831
I20260619 20:31:03.232295 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 7427
I20260619 20:31:03.232411 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:31:03.232637 22108 external_mini_cluster.cc:1428] Setting key 7bd7e0ea09aad961471784048af551b7
I20260619 20:31:03.233522 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33501
--builtin_ntp_servers=127.21.151.20:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:31:03.313302 7558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:03.313503 7558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:03.313524 7558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:03.314661 7558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:03.314698 7558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:03.314723 7558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:31:03.316130 7558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:03.316378 7558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:03.316671 7558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:03.321473 7564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:03.321465 7566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:03.321461 7563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:03.321767 7558 server_base.cc:1034] running on GCE node
I20260619 20:31:03.321987 7558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:03.322222 7558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:03.323413 7558 hybrid_clock.cc:648] HybridClock initialized: now 1781901063323401 us; error 40 us; skew 500 ppm
I20260619 20:31:03.324779 7558 webserver.cc:460] Webserver started at http://127.21.151.3:46263/ using document root <none> and password file <none>
I20260619 20:31:03.325055 7558 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:03.325132 7558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:03.325318 7558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:03.326416 7558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "e8a12e44e54e4c4ba89253ff947c4ea6"
format_stamp: "Formatted at 2026-06-19 20:31:03 on dist-test-slave-f9rk"
server_key: "dc70e6eb26d9fe8356112c3ccb054fa5"
server_key_iv: "cc69156a2d13d60e70d4d412cff3abb4"
server_key_version: "encryptionkey@0"
I20260619 20:31:03.326797 7558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "e8a12e44e54e4c4ba89253ff947c4ea6"
format_stamp: "Formatted at 2026-06-19 20:31:03 on dist-test-slave-f9rk"
server_key: "dc70e6eb26d9fe8356112c3ccb054fa5"
server_key_iv: "cc69156a2d13d60e70d4d412cff3abb4"
server_key_version: "encryptionkey@0"
I20260619 20:31:03.328509 7558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 20:31:03.329334 7572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:03.329512 7558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:03.329613 7558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "e8a12e44e54e4c4ba89253ff947c4ea6"
format_stamp: "Formatted at 2026-06-19 20:31:03 on dist-test-slave-f9rk"
server_key: "dc70e6eb26d9fe8356112c3ccb054fa5"
server_key_iv: "cc69156a2d13d60e70d4d412cff3abb4"
server_key_version: "encryptionkey@0"
I20260619 20:31:03.329700 7558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:03.354661 7558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:03.355005 7558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:03.355167 7558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:03.355424 7558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:03.355837 7558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:03.355906 7558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:03.355973 7558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:03.356005 7558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:03.362062 7558 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:42491
I20260619 20:31:03.362103 7685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:42491 every 8 connection(s)
I20260619 20:31:03.362471 7558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:31:03.368132 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.363236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:34361 (local address 127.21.151.62:33501)
0619 20:31:03.363283 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:03.363285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:03.363612 (+ 327us) server_negotiation.cc:407] Connection header received
0619 20:31:03.364214 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:03.364215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:03.364229 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:03.364300 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:03.365016 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.365538 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.366596 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.366794 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.366904 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.367814 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:03.367816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:03.367816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:03.367822 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:03.367836 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:03.367840 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:03.367912 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:03.368025 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 20:31:03.368052 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:31:03.368363 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.363228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33501 (local address 127.21.151.3:34361)
0619 20:31:03.363461 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:03.363507 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:03.364081 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:03.364377 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:03.364382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:03.364392 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:03.364906 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.364911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.365641 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.365642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.366478 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.366481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.366914 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.366915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.367060 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.367587 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:31:03.367588 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:03.367676 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:31:03.367710 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:03.367912 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:03.367927 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:03.367930 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:03.368015 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 20:31:03.368119 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
I20260619 20:31:03.368369 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 7558
I20260619 20:31:03.368487 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:31:03.368682 22108 external_mini_cluster.cc:1428] Setting key f65accc10cf3d4a97c3b0616e12f658f
I20260619 20:31:03.368798 7686 heartbeater.cc:347] Connected to a master server at 127.21.151.62:33501
I20260619 20:31:03.368901 7686 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:03.369124 7686 heartbeater.cc:511] Master 127.21.151.62:33501 requested a full tablet report, sending...
I20260619 20:31:03.369537 7236 ts_manager.cc:194] Registered new tserver with Master: e8a12e44e54e4c4ba89253ff947c4ea6 (127.21.151.3:42491)
I20260619 20:31:03.369880 7236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.3:34361
I20260619 20:31:03.380650 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:31:03.384586 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.382046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.21.151.62:33501)
0619 20:31:03.382086 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:03.382088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:03.382161 (+ 73us) server_negotiation.cc:407] Connection header received
0619 20:31:03.382233 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:03.382234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:03.382242 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:03.382298 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:03.382428 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.382869 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.383576 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.383812 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.383867 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.384410 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:03.384411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:03.384411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:03.384415 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:03.384434 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:03.384438 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:03.384474 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:03.384512 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 20:31:03.384531 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:31:03.386044 7236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:31:03.387542 7236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:31:03.393843 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.389587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:34049 (local address 127.0.0.1:53058)
0619 20:31:03.389906 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:03.389917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:03.389970 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:03.390194 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:03.390197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:03.390199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:03.390304 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.390306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.391704 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.391705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.392793 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.392796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.393633 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.393633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.393689 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.393699 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 20:31:03.393754 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:31:03.393789 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":98,"spinlock_wait_cycles":768,"thread_start_us":104,"threads_started":1}
I20260619 20:31:03.393934 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.389817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.21.151.2:34049)
0619 20:31:03.390001 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:03.390006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:03.390017 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:31:03.390062 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:03.390065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:03.390081 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:03.390175 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:03.390405 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.391590 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.392885 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.393552 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.393628 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.393717 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:03.393781 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 20:31:03.393817 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260619 20:31:03.394336 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.389587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:46005 (local address 127.0.0.1:44584)
0619 20:31:03.390265 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:03.390274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:03.390354 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:03.390625 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:03.390626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:03.390627 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:03.390668 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.390670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.391669 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.391671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.393025 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.393028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.394099 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.394100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.394174 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.394179 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:31:03.394231 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 20:31:03.394265 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":67,"thread_start_us":89,"threads_started":1}
I20260619 20:31:03.394563 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.389817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.21.151.1:46005)
0619 20:31:03.389985 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:03.389990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:03.390353 (+ 363us) server_negotiation.cc:407] Connection header received
0619 20:31:03.390444 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:03.390446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:03.390479 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:03.390553 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:03.390764 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.391555 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.393117 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.393850 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.393916 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.393997 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:03.394447 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 20:31:03.394490 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260619 20:31:03.394785 7489 tablet_service.cc:1467] Processing CreateTablet for tablet e9bf8ac373f7401784ab691ec30f0efa (DEFAULT_TABLE table=test-table [id=95c1ab8a7b1c4a35bf1ef08b49521f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:03.395102 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.389550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:42491 (local address 127.0.0.1:40368)
0619 20:31:03.389664 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:03.389705 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:03.389767 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:03.390194 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:03.390197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:03.390201 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:03.390304 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.390306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.391745 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.391746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.393025 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.393028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.394898 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.394899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.394965 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.394968 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:03.394998 (+ 30us) client_negotiation.cc:240] Negotiation successful
0619 20:31:03.395037 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":121,"thread_start_us":67,"threads_started":1}
I20260619 20:31:03.395170 7489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bf8ac373f7401784ab691ec30f0efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:03.395166 7358 tablet_service.cc:1467] Processing CreateTablet for tablet e9bf8ac373f7401784ab691ec30f0efa (DEFAULT_TABLE table=test-table [id=95c1ab8a7b1c4a35bf1ef08b49521f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:03.395490 7358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bf8ac373f7401784ab691ec30f0efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:03.395534 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.389718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.21.151.3:42491)
0619 20:31:03.390027 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:03.390031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:03.390040 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:31:03.390072 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:03.390075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:03.390088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:03.390175 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:03.390503 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.391633 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.394032 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.394803 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.395286 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.395386 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:03.395412 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 20:31:03.395452 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":99,"threads_started":1}
I20260619 20:31:03.396098 7620 tablet_service.cc:1467] Processing CreateTablet for tablet e9bf8ac373f7401784ab691ec30f0efa (DEFAULT_TABLE table=test-table [id=95c1ab8a7b1c4a35bf1ef08b49521f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:03.396389 7620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bf8ac373f7401784ab691ec30f0efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:03.398675 7700 tablet_bootstrap.cc:492] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: Bootstrap starting.
I20260619 20:31:03.399106 7702 tablet_bootstrap.cc:492] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: Bootstrap starting.
I20260619 20:31:03.399434 7700 tablet_bootstrap.cc:654] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:03.399564 7701 tablet_bootstrap.cc:492] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11: Bootstrap starting.
I20260619 20:31:03.399816 7702 tablet_bootstrap.cc:654] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:03.399801 7700 log.cc:824] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:03.400171 7702 log.cc:824] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:03.400213 7701 tablet_bootstrap.cc:654] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:03.400527 7701 log.cc:824] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:03.400707 7700 tablet_bootstrap.cc:492] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: No bootstrap required, opened a new log
I20260619 20:31:03.400776 7700 ts_tablet_manager.cc:1397] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 20:31:03.400888 7702 tablet_bootstrap.cc:492] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: No bootstrap required, opened a new log
I20260619 20:31:03.400970 7702 ts_tablet_manager.cc:1397] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:31:03.401353 7701 tablet_bootstrap.cc:492] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11: No bootstrap required, opened a new log
I20260619 20:31:03.401466 7701 ts_tablet_manager.cc:1397] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:31:03.403136 7702 raft_consensus.cc:348] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 } } peers { permanent_uuid: "05ee3a84af5e4366b3154c6078ec4cec" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34049 } } peers { permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 } }
I20260619 20:31:03.403265 7702 raft_consensus.cc:374] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:03.403286 7702 raft_consensus.cc:729] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a12e44e54e4c4ba89253ff947c4ea6, State: Initialized, Role: FOLLOWER
I20260619 20:31:03.403448 7702 consensus_queue.cc:260] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 } } peers { permanent_uuid: "05ee3a84af5e4366b3154c6078ec4cec" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34049 } } peers { permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 } }
I20260619 20:31:03.403460 7700 raft_consensus.cc:348] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 } } peers { permanent_uuid: "05ee3a84af5e4366b3154c6078ec4cec" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34049 } } peers { permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 } }
I20260619 20:31:03.403648 7700 raft_consensus.cc:374] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:03.403705 7700 raft_consensus.cc:729] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ee3a84af5e4366b3154c6078ec4cec, State: Initialized, Role: FOLLOWER
I20260619 20:31:03.403735 7702 ts_tablet_manager.cc:1428] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:31:03.403851 7686 heartbeater.cc:503] Master 127.21.151.62:33501 was elected leader, sending a full tablet report...
I20260619 20:31:03.403844 7700 consensus_queue.cc:260] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 } } peers { permanent_uuid: "05ee3a84af5e4366b3154c6078ec4cec" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34049 } } peers { permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 } }
I20260619 20:31:03.404088 7700 ts_tablet_manager.cc:1428] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:31:03.404117 7555 heartbeater.cc:503] Master 127.21.151.62:33501 was elected leader, sending a full tablet report...
I20260619 20:31:03.404799 7701 raft_consensus.cc:348] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 } } peers { permanent_uuid: "05ee3a84af5e4366b3154c6078ec4cec" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34049 } } peers { permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 } }
I20260619 20:31:03.404959 7701 raft_consensus.cc:374] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:03.405009 7701 raft_consensus.cc:729] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e88ab4d14034a68b6092151ec0bbb11, State: Initialized, Role: FOLLOWER
I20260619 20:31:03.405123 7701 consensus_queue.cc:260] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 } } peers { permanent_uuid: "05ee3a84af5e4366b3154c6078ec4cec" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34049 } } peers { permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 } }
I20260619 20:31:03.405339 7701 ts_tablet_manager.cc:1428] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 20:31:03.405417 7424 heartbeater.cc:503] Master 127.21.151.62:33501 was elected leader, sending a full tablet report...
W20260619 20:31:03.475560 7556 tablet.cc:2369] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:31:03.530315 7707 raft_consensus.cc:482] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:31:03.530473 7707 raft_consensus.cc:504] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 } } peers { permanent_uuid: "05ee3a84af5e4366b3154c6078ec4cec" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34049 } } peers { permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 } }
I20260619 20:31:03.530869 7707 leader_election.cc:288] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8a12e44e54e4c4ba89253ff947c4ea6 (127.21.151.3:42491), 7e88ab4d14034a68b6092151ec0bbb11 (127.21.151.1:46005)
I20260619 20:31:03.533936 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.531136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:53363 (local address 127.21.151.3:42491)
0619 20:31:03.531183 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:03.531185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:03.531330 (+ 145us) server_negotiation.cc:407] Connection header received
0619 20:31:03.531361 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:03.531361 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:03.531371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:03.531442 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:03.531607 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.532131 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.533185 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.533602 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.533680 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.533689 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:03.533823 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 20:31:03.533863 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:31:03.533950 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.531143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:43183 (local address 127.21.151.1:46005)
0619 20:31:03.531192 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:03.531194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:03.531206 (+ 12us) server_negotiation.cc:407] Connection header received
0619 20:31:03.531262 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:03.531263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:03.531275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:03.531328 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:03.531510 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.532040 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.533078 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.533597 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.533685 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.533693 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:03.533853 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 20:31:03.533890 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:31:03.533955 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.531086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:42491 (local address 127.21.151.2:53363)
0619 20:31:03.531244 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:03.531253 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:03.531314 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:03.531446 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:03.531447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:03.531448 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:03.531534 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.531535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.532211 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.532212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.533105 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.533107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.533674 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.533675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.533756 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.533759 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:03.533801 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 20:31:03.533863 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260619 20:31:03.533970 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.531061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:46005 (local address 127.21.151.2:43183)
0619 20:31:03.531128 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:03.531139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:03.531192 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:03.531348 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:03.531349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:03.531350 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:03.531425 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.531428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.532126 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.532127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.532997 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:03.532999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.533696 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.533698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:03.533764 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.533767 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:03.533838 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 20:31:03.533875 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":13}
I20260619 20:31:03.534305 7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bf8ac373f7401784ab691ec30f0efa" candidate_uuid: "05ee3a84af5e4366b3154c6078ec4cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e88ab4d14034a68b6092151ec0bbb11" is_pre_election: true
I20260619 20:31:03.534314 7640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bf8ac373f7401784ab691ec30f0efa" candidate_uuid: "05ee3a84af5e4366b3154c6078ec4cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" is_pre_election: true
I20260619 20:31:03.534457 7640 raft_consensus.cc:2454] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05ee3a84af5e4366b3154c6078ec4cec in term 0.
I20260619 20:31:03.534471 7378 raft_consensus.cc:2454] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05ee3a84af5e4366b3154c6078ec4cec in term 0.
I20260619 20:31:03.534683 7444 leader_election.cc:302] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ee3a84af5e4366b3154c6078ec4cec, e8a12e44e54e4c4ba89253ff947c4ea6; no voters:
I20260619 20:31:03.534853 7707 raft_consensus.cc:2781] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:31:03.534909 7707 raft_consensus.cc:482] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:31:03.534926 7707 raft_consensus.cc:3037] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:03.535874 7707 raft_consensus.cc:504] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 } } peers { permanent_uuid: "05ee3a84af5e4366b3154c6078ec4cec" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34049 } } peers { permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 } }
I20260619 20:31:03.536006 7707 leader_election.cc:288] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [CANDIDATE]: Term 1 election: Requested vote from peers e8a12e44e54e4c4ba89253ff947c4ea6 (127.21.151.3:42491), 7e88ab4d14034a68b6092151ec0bbb11 (127.21.151.1:46005)
I20260619 20:31:03.536145 7640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bf8ac373f7401784ab691ec30f0efa" candidate_uuid: "05ee3a84af5e4366b3154c6078ec4cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6"
I20260619 20:31:03.536161 7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bf8ac373f7401784ab691ec30f0efa" candidate_uuid: "05ee3a84af5e4366b3154c6078ec4cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e88ab4d14034a68b6092151ec0bbb11"
I20260619 20:31:03.536242 7640 raft_consensus.cc:3037] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:03.536267 7378 raft_consensus.cc:3037] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:03.536984 7640 raft_consensus.cc:2454] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ee3a84af5e4366b3154c6078ec4cec in term 1.
I20260619 20:31:03.537052 7378 raft_consensus.cc:2454] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ee3a84af5e4366b3154c6078ec4cec in term 1.
I20260619 20:31:03.537169 7444 leader_election.cc:302] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ee3a84af5e4366b3154c6078ec4cec, e8a12e44e54e4c4ba89253ff947c4ea6; no voters:
I20260619 20:31:03.537268 7707 raft_consensus.cc:2781] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:03.537416 7707 raft_consensus.cc:686] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 1 LEADER]: Becoming Leader. State: Replica: 05ee3a84af5e4366b3154c6078ec4cec, State: Running, Role: LEADER
I20260619 20:31:03.537492 7707 consensus_queue.cc:237] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 } } peers { permanent_uuid: "05ee3a84af5e4366b3154c6078ec4cec" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34049 } } peers { permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 } }
I20260619 20:31:03.538301 7235 catalog_manager.cc:5495] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec reported cstate change: term changed from 0 to 1, leader changed from <none> to 05ee3a84af5e4366b3154c6078ec4cec (127.21.151.2). New cstate: current_term: 1 leader_uuid: "05ee3a84af5e4366b3154c6078ec4cec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05ee3a84af5e4366b3154c6078ec4cec" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 } health_report { overall_health: UNKNOWN } } }
I20260619 20:31:03.575573 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.573476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.21.151.2:34049)
0619 20:31:03.573536 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:03.573538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:03.573547 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:31:03.573599 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:03.573599 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:03.573617 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:03.573686 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:03.573780 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.574228 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.574897 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:03.575073 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:03.575120 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:03.575246 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:03.575340 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:03.575397 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:03.575433 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 20:31:03.575449 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 20:31:03.577350 7378 raft_consensus.cc:1261] T e9bf8ac373f7401784ab691ec30f0efa P 7e88ab4d14034a68b6092151ec0bbb11 [term 1 FOLLOWER]: Refusing update from remote peer 05ee3a84af5e4366b3154c6078ec4cec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:03.577339 7640 raft_consensus.cc:1261] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6 [term 1 FOLLOWER]: Refusing update from remote peer 05ee3a84af5e4366b3154c6078ec4cec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:03.577728 7707 consensus_queue.cc:1035] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a12e44e54e4c4ba89253ff947c4ea6" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:03.577893 7710 consensus_queue.cc:1035] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e88ab4d14034a68b6092151ec0bbb11" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:03.580382 7714 mvcc.cc:205] Tried to move back new op lower bound from 7298666756410118144 to 7298666756250202112. Current Snapshot: MvccSnapshot[applied={T|T < 7298666756410118144}]
I20260619 20:31:03.581352 7715 mvcc.cc:205] Tried to move back new op lower bound from 7298666756410118144 to 7298666756250202112. Current Snapshot: MvccSnapshot[applied={T|T < 7298666756410118144 or (T in {7298666756410118144})}]
I20260619 20:31:03.581470 7713 mvcc.cc:205] Tried to move back new op lower bound from 7298666756410118144 to 7298666756250202112. Current Snapshot: MvccSnapshot[applied={T|T < 7298666756410118144 or (T in {7298666756410118144})}]
I20260619 20:31:03.582418 7235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:03.582507 7235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:03.583235 7235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60493796799d49bdb8d89d7efbcdd148: Sending DeleteTablet for 3 replicas of tablet e9bf8ac373f7401784ab691ec30f0efa
I20260619 20:31:03.583544 7620 tablet_service.cc:1514] Processing DeleteTablet for tablet e9bf8ac373f7401784ab691ec30f0efa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:03 UTC) from {username='slave'} at 127.0.0.1:40368
I20260619 20:31:03.583621 7358 tablet_service.cc:1514] Processing DeleteTablet for tablet e9bf8ac373f7401784ab691ec30f0efa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:03 UTC) from {username='slave'} at 127.0.0.1:44584
I20260619 20:31:03.583815 7723 tablet_replica.cc:330] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: stopping tablet replica
I20260619 20:31:03.583926 7723 raft_consensus.cc:2229] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:31:03.584035 7723 raft_consensus.cc:2258] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:03.584132 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 7296
I20260619 20:31:03.584959 7723 ts_tablet_manager.cc:1905] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:03.586145 7489 tablet_service.cc:1514] Processing DeleteTablet for tablet e9bf8ac373f7401784ab691ec30f0efa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:03 UTC) from {username='slave'} at 127.0.0.1:53058
I20260619 20:31:03.586877 7723 ts_tablet_manager.cc:1918] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:03.586951 7723 log.cc:1196] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/e9bf8ac373f7401784ab691ec30f0efa
I20260619 20:31:03.587173 7723 ts_tablet_manager.cc:1939] T e9bf8ac373f7401784ab691ec30f0efa P e8a12e44e54e4c4ba89253ff947c4ea6: Deleting consensus metadata
I20260619 20:31:03.587368 7725 tablet_replica.cc:330] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: stopping tablet replica
I20260619 20:31:03.587461 7725 raft_consensus.cc:2229] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:31:03.587550 7725 raft_consensus.cc:2258] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:03.587771 7222 catalog_manager.cc:4841] TS e8a12e44e54e4c4ba89253ff947c4ea6 (127.21.151.3:42491): tablet e9bf8ac373f7401784ab691ec30f0efa (table test-table [id=95c1ab8a7b1c4a35bf1ef08b49521f2f]) successfully deleted
I20260619 20:31:03.588050 7725 ts_tablet_manager.cc:1905] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:03.589969 7725 ts_tablet_manager.cc:1918] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:03.590040 7725 log.cc:1196] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/e9bf8ac373f7401784ab691ec30f0efa
W20260619 20:31:03.590066 7221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:46005 (error 108)
I20260619 20:31:03.590315 7725 ts_tablet_manager.cc:1939] T e9bf8ac373f7401784ab691ec30f0efa P 05ee3a84af5e4366b3154c6078ec4cec: Deleting consensus metadata
I20260619 20:31:03.590523 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:03.590409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:46005 (local address 127.0.0.1:44594)
0619 20:31:03.590451 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:03.590492 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:46005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 20:31:03.590636 7221 catalog_manager.cc:4584] TS 7e88ab4d14034a68b6092151ec0bbb11 (127.21.151.1:46005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9bf8ac373f7401784ab691ec30f0efa: Network error: Client connection negotiation failed: client connection to 127.21.151.1:46005: connect: Connection refused (error 111)
I20260619 20:31:03.590749 7220 catalog_manager.cc:4841] TS 05ee3a84af5e4366b3154c6078ec4cec (127.21.151.2:34049): tablet e9bf8ac373f7401784ab691ec30f0efa (table test-table [id=95c1ab8a7b1c4a35bf1ef08b49521f2f]) successfully deleted
I20260619 20:31:03.591012 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 7427
I20260619 20:31:03.596335 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 7558
I20260619 20:31:03.600551 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 7205
2026-06-19T20:31:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (816 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:03 dist-test-slave-f9rk krb5kdc[7729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:03 dist-test-slave-f9rk krb5kdc[7729](info): set up 2 sockets
Jun 19 20:31:03 dist-test-slave-f9rk krb5kdc[7729](info): commencing operation
krb5kdc: starting...
W20260619 20:31:04.616353 22108 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:04 dist-test-slave-f9rk krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:04.730997 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:38413
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:38413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:04.811203 7742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:04.811419 7742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:04.811442 7742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:04.812729 7742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:04.812780 7742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:04.812795 7742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:04.812808 7742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:04.812820 7742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:04.814309 7742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:38413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:38413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:04.814564 7742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:04.814844 7742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:04.819929 7747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:04.819927 7748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:04.819942 7750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:04.820122 7742 server_base.cc:1034] running on GCE node
I20260619 20:31:04.820425 7742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:04.820752 7742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:04.821928 7742 hybrid_clock.cc:648] HybridClock initialized: now 1781901064821913 us; error 51 us; skew 500 ppm
Jun 19 20:31:04 dist-test-slave-f9rk krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901064, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:04.831297 7742 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:04.831880 7742 webserver.cc:460] Webserver started at http://127.21.151.62:46585/ using document root <none> and password file <none>
I20260619 20:31:04.832146 7742 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:04.832218 7742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:04.832370 7742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:04.833475 7742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "69b5ec2894f7402b9637a17d04732754"
format_stamp: "Formatted at 2026-06-19 20:31:04 on dist-test-slave-f9rk"
server_key: "04a9166584e617f900307cd623e4afb6"
server_key_iv: "788ee6a2f4398bcf54a1c78659c28c0a"
server_key_version: "encryptionkey@0"
I20260619 20:31:04.833849 7742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "69b5ec2894f7402b9637a17d04732754"
format_stamp: "Formatted at 2026-06-19 20:31:04 on dist-test-slave-f9rk"
server_key: "04a9166584e617f900307cd623e4afb6"
server_key_iv: "788ee6a2f4398bcf54a1c78659c28c0a"
server_key_version: "encryptionkey@0"
I20260619 20:31:04.835220 7742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:31:04.836115 7757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:04.836295 7742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:04.836403 7742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "69b5ec2894f7402b9637a17d04732754"
format_stamp: "Formatted at 2026-06-19 20:31:04 on dist-test-slave-f9rk"
server_key: "04a9166584e617f900307cd623e4afb6"
server_key_iv: "788ee6a2f4398bcf54a1c78659c28c0a"
server_key_version: "encryptionkey@0"
I20260619 20:31:04.836493 7742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:04.853368 7742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:04.855878 7742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:04.856045 7742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:04.859953 7742 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:38413
I20260619 20:31:04.860005 7810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:38413 every 8 connection(s)
I20260619 20:31:04.860378 7742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:04.861260 7811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:04.863502 7811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754: Bootstrap starting.
I20260619 20:31:04.864107 7811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:04.864398 7811 log.cc:824] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:04.865077 7811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754: No bootstrap required, opened a new log
I20260619 20:31:04.865736 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 7742
I20260619 20:31:04.865856 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:04.866050 22108 external_mini_cluster.cc:1428] Setting key 2e833c4faecc3dd32a1a56fc09ce859c
I20260619 20:31:04.867148 7811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b5ec2894f7402b9637a17d04732754" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38413 } }
I20260619 20:31:04.867307 7811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:04.867372 7811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b5ec2894f7402b9637a17d04732754, State: Initialized, Role: FOLLOWER
I20260619 20:31:04.867547 7811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b5ec2894f7402b9637a17d04732754" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38413 } }
I20260619 20:31:04.867671 7811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:04.867713 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:04.867763 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:04.868594 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b5ec2894f7402b9637a17d04732754" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38413 } }
I20260619 20:31:04.868678 7811 leader_election.cc:302] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69b5ec2894f7402b9637a17d04732754; no voters:
I20260619 20:31:04.868865 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:04.868899 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:04.869062 7816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [term 1 LEADER]: Becoming Leader. State: Replica: 69b5ec2894f7402b9637a17d04732754, State: Running, Role: LEADER
I20260619 20:31:04.869138 7811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:04.869191 7816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b5ec2894f7402b9637a17d04732754" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38413 } }
I20260619 20:31:04.870711 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69b5ec2894f7402b9637a17d04732754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b5ec2894f7402b9637a17d04732754" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38413 } } }
I20260619 20:31:04.870759 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69b5ec2894f7402b9637a17d04732754. Latest consensus state: current_term: 1 leader_uuid: "69b5ec2894f7402b9637a17d04732754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b5ec2894f7402b9637a17d04732754" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38413 } } }
I20260619 20:31:04.870855 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:04.870868 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:04.871066 7831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:04.871546 7831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:04.872951 7831 catalog_manager.cc:1347] Generated new cluster ID: 57065d7da11440c696c5b43ebd965a6c
I20260619 20:31:04.873004 7831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:31:04 dist-test-slave-f9rk krb5kdc[7729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:04.877434 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:04.866575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.21.151.62:38413)
0619 20:31:04.866750 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:04.866757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:04.866782 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:31:04.867134 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:04.867139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:04.867191 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:04.867331 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:04.868117 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:04.869217 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:04.870091 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:04.870728 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:04.870866 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:04.873807 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:04.873811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:04.873811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:04.873823 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:04.875829 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:04.876242 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:04.876244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:04.876245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:04.876370 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:04.876579 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:04.876580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:04.876580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:04.876854 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:04.877005 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:04.877112 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 20:31:04.877239 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260619 20:31:04.888020 7831 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:04.888721 7831 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:04.892831 7831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754: Generated new TSK 0
I20260619 20:31:04.893007 7831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:31:04.917706 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38413
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:04.998237 7837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:04.998436 7837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:04.998459 7837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:04.999706 7837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:04.999745 7837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:04.999773 7837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:31:05.001261 7837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:05.001498 7837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:05.001739 7837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:05.006641 7843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:05.006685 7842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:05.006641 7845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:05.006772 7837 server_base.cc:1034] running on GCE node
I20260619 20:31:05.007066 7837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:05.007259 7837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:05.008395 7837 hybrid_clock.cc:648] HybridClock initialized: now 1781901065008379 us; error 34 us; skew 500 ppm
Jun 19 20:31:05 dist-test-slave-f9rk krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901065, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:05.010730 7837 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:31:05.011126 7837 webserver.cc:460] Webserver started at http://127.21.151.1:45497/ using document root <none> and password file <none>
I20260619 20:31:05.011317 7837 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:05.011360 7837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:05.011448 7837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:05.012584 7837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "48d61ccfd39c4e529c18edf3f75b3fd9"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "d3124151351239ad8c5cecc5e0e334c8"
server_key_iv: "4090a748011544d82757f0fc86b83306"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.012892 7837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "48d61ccfd39c4e529c18edf3f75b3fd9"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "d3124151351239ad8c5cecc5e0e334c8"
server_key_iv: "4090a748011544d82757f0fc86b83306"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.014097 7837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:31:05.014767 7852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.014921 7837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:05.015007 7837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "48d61ccfd39c4e529c18edf3f75b3fd9"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "d3124151351239ad8c5cecc5e0e334c8"
server_key_iv: "4090a748011544d82757f0fc86b83306"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.015061 7837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:05.027338 7837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:05.029241 7837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:05.029356 7837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:05.029543 7837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:05.029808 7837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:05.029847 7837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.029870 7837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:05.029883 7837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.035542 7837 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:39719
I20260619 20:31:05.035617 7965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:39719 every 8 connection(s)
I20260619 20:31:05.035996 7837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
Jun 19 20:31:05 dist-test-slave-f9rk krb5kdc[7729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901065, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:05.042562 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 7837
I20260619 20:31:05.042703 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:31:05.042955 22108 external_mini_cluster.cc:1428] Setting key f9386b7b1f381387a676c6efcac91ee2
I20260619 20:31:05.045689 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.036737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:36731 (local address 127.21.151.62:38413)
0619 20:31:05.036806 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.036808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.037010 (+ 202us) server_negotiation.cc:407] Connection header received
0619 20:31:05.037481 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.037482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.037491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.037551 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:05.038459 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.039033 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.040068 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.040319 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.040379 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.042425 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:05.042427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:05.042427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:05.042437 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:05.044285 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:05.044670 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:05.044671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:05.044672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:05.044748 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:05.045003 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:05.045004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:05.045005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:05.045131 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:05.045211 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.045491 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.045597 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:31:05.045871 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.036697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38413 (local address 127.21.151.1:36731)
0619 20:31:05.036881 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:05.036922 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:05.037399 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:05.037615 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:05.037618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:05.038178 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:05.038375 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.038380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.039145 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.039146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.039995 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.039997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.040394 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.040395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.040513 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.040994 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:05.041002 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:05.042338 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:05.044367 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:05.044373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:05.044374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:05.044595 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:05.044814 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:05.044815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:05.044815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:05.044926 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:05.045201 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:05.045208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:05.045352 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 20:31:05.045469 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 20:31:05.045626 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":116,"threads_started":1}
I20260619 20:31:05.046582 7966 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38413
I20260619 20:31:05.046663 7966 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:05.046845 7966 heartbeater.cc:511] Master 127.21.151.62:38413 requested a full tablet report, sending...
I20260619 20:31:05.047385 7774 ts_manager.cc:194] Registered new tserver with Master: 48d61ccfd39c4e529c18edf3f75b3fd9 (127.21.151.1:39719)
I20260619 20:31:05.048081 7774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:36731
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:31:05.067103 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38413
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:05.150804 7971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:05.150995 7971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:05.151017 7971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:05.152259 7971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:05.152300 7971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:05.152356 7971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:31:05.153779 7971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:05.154012 7971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:05.154227 7971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:05.159061 7976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:05.159103 7979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:05.159072 7977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:05.159315 7971 server_base.cc:1034] running on GCE node
I20260619 20:31:05.159511 7971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:05.159770 7971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:05.160913 7971 hybrid_clock.cc:648] HybridClock initialized: now 1781901065160897 us; error 31 us; skew 500 ppm
Jun 19 20:31:05 dist-test-slave-f9rk krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901065, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:05.163220 7971 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:31:05.163693 7971 webserver.cc:460] Webserver started at http://127.21.151.2:42027/ using document root <none> and password file <none>
I20260619 20:31:05.163931 7971 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:05.164036 7971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:05.164180 7971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:05.165226 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "eb07ead79c4f4b49a11907a2c7609c80"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "854d31462fe0b4e080c95e6ed4c966b9"
server_key_iv: "9ec11133a6aab15cb2e4410ab6d82def"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.165596 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "eb07ead79c4f4b49a11907a2c7609c80"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "854d31462fe0b4e080c95e6ed4c966b9"
server_key_iv: "9ec11133a6aab15cb2e4410ab6d82def"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.166896 7971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:31:05.167676 7986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.167850 7971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:31:05.167944 7971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "eb07ead79c4f4b49a11907a2c7609c80"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "854d31462fe0b4e080c95e6ed4c966b9"
server_key_iv: "9ec11133a6aab15cb2e4410ab6d82def"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.168023 7971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:05.184743 7971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:05.187602 7971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:05.187774 7971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:05.188032 7971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:05.188422 7971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:05.188450 7971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.188472 7971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:05.188483 7971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.196642 7971 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:44009
I20260619 20:31:05.196677 8099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:44009 every 8 connection(s)
I20260619 20:31:05.197041 7971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:31:05.203748 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 7971
Jun 19 20:31:05 dist-test-slave-f9rk krb5kdc[7729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901065, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:05.203871 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:31:05.204090 22108 external_mini_cluster.cc:1428] Setting key af671b6c05ca9ecaaae37444fee34c93
I20260619 20:31:05.207468 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.197753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:53457 (local address 127.21.151.62:38413)
0619 20:31:05.197795 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.197797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.198007 (+ 210us) server_negotiation.cc:407] Connection header received
0619 20:31:05.198494 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.198495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.198505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.198564 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:05.199497 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.200114 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.201284 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.201570 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.201660 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.204473 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:05.204475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:05.204475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:05.204484 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:05.205978 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:05.206438 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:05.206440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:05.206441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:05.206502 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:05.206767 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:05.206768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:05.206768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:05.206918 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:05.206994 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.207283 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.207372 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:31:05.207732 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.197717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38413 (local address 127.21.151.2:53457)
0619 20:31:05.197893 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:05.197934 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:05.198391 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:05.198634 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:05.198650 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:05.199202 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:05.199415 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.199419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.200197 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.200197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.201194 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.201197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.201692 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.201693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.201842 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.202452 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:05.202462 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:05.204332 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:05.206095 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:05.206102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:05.206105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:05.206343 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:05.206575 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:05.206576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:05.206577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:05.206696 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:05.206996 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:05.207003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:05.207149 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 20:31:05.207262 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 20:31:05.207436 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260619 20:31:05.208141 8100 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38413
I20260619 20:31:05.208231 8100 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:05.208431 8100 heartbeater.cc:511] Master 127.21.151.62:38413 requested a full tablet report, sending...
I20260619 20:31:05.208792 7774 ts_manager.cc:194] Registered new tserver with Master: eb07ead79c4f4b49a11907a2c7609c80 (127.21.151.2:44009)
I20260619 20:31:05.209172 7774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:53457
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:31:05.228896 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38413
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:36963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:05.315985 8105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:05.316234 8105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:05.316274 8105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:05.317499 8105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:05.317564 8105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:05.317615 8105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:31:05.319128 8105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:05.319388 8105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:05.319659 8105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:05.324558 8113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:05.324613 8110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:05.324558 8111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:05.324760 8105 server_base.cc:1034] running on GCE node
I20260619 20:31:05.324987 8105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:05.325253 8105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:05.326417 8105 hybrid_clock.cc:648] HybridClock initialized: now 1781901065326405 us; error 35 us; skew 500 ppm
Jun 19 20:31:05 dist-test-slave-f9rk krb5kdc[7729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901065, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:05.328855 8105 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:31:05.329263 8105 webserver.cc:460] Webserver started at http://127.21.151.3:38145/ using document root <none> and password file <none>
I20260619 20:31:05.329496 8105 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:05.329564 8105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:05.329711 8105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:05.330736 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "333d29cb808547529acce01628b4436f"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "5b0a0d647ba206aaa9d568a621807253"
server_key_iv: "fe320935d81d28e3868eb07b4774d95e"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.331084 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "333d29cb808547529acce01628b4436f"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "5b0a0d647ba206aaa9d568a621807253"
server_key_iv: "fe320935d81d28e3868eb07b4774d95e"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.332433 8105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:31:05.333104 8120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.333266 8105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:05.333376 8105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "333d29cb808547529acce01628b4436f"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "5b0a0d647ba206aaa9d568a621807253"
server_key_iv: "fe320935d81d28e3868eb07b4774d95e"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.333451 8105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:05.353754 8105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:05.355736 8105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:05.355893 8105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:05.356125 8105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:05.356439 8105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:05.356499 8105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.356544 8105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:05.356570 8105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.362267 8105 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:39467
I20260619 20:31:05.362339 8233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:39467 every 8 connection(s)
I20260619 20:31:05.362674 8105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:31:05.366714 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 8105
I20260619 20:31:05.366843 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:31:05.367043 22108 external_mini_cluster.cc:1428] Setting key 7120274e51882c8083ff428c0baa5879
Jun 19 20:31:05 dist-test-slave-f9rk krb5kdc[7729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901065, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:05.372318 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.363381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:49369 (local address 127.21.151.62:38413)
0619 20:31:05.363427 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.363429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.363657 (+ 228us) server_negotiation.cc:407] Connection header received
0619 20:31:05.364149 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.364150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.364160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.364210 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:05.365116 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.365625 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.366627 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.366806 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.366896 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.369388 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:05.369389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:05.369389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:05.369396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:05.370968 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:05.371334 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:05.371335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:05.371335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:05.371399 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:05.371661 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:05.371662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:05.371662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:05.371786 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:05.371845 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.372190 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.372254 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:31:05.372527 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.363357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:38413 (local address 127.21.151.3:49369)
0619 20:31:05.363500 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:05.363534 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:05.364059 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:05.364263 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:05.364266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:05.364803 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:05.365030 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.365034 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.365703 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.365704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.366545 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.366547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.366932 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.366933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.367092 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.367729 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:05.367739 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:05.369308 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:05.371046 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:05.371052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:05.371053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:05.371267 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:05.371474 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:05.371475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:05.371475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:05.371572 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:05.371841 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:05.371845 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:05.372082 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 20:31:05.372171 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 20:31:05.372295 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260619 20:31:05.373001 8234 heartbeater.cc:347] Connected to a master server at 127.21.151.62:38413
I20260619 20:31:05.373085 8234 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:05.373219 8234 heartbeater.cc:511] Master 127.21.151.62:38413 requested a full tablet report, sending...
I20260619 20:31:05.373553 7774 ts_manager.cc:194] Registered new tserver with Master: 333d29cb808547529acce01628b4436f (127.21.151.3:39467)
I20260619 20:31:05.373878 7774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:49369
I20260619 20:31:05.378804 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:31:05.386435 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.380111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.21.151.62:38413)
0619 20:31:05.380191 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.380194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.380311 (+ 117us) server_negotiation.cc:407] Connection header received
0619 20:31:05.380386 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.380386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.380395 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.380466 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:05.381004 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.381488 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.382208 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.382429 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.382506 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.383878 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:05.383879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:05.383879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:05.383887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:05.385191 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:05.385585 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:05.385586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:05.385587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:05.385650 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:05.385872 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:05.385873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:05.385874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:05.386018 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:05.386136 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.386260 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.386326 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:31:05.387943 7774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:31:05.389501 7774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:31:05.395757 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.391742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:39467 (local address 127.0.0.1:36826)
0619 20:31:05.392246 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:05.392255 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:05.392308 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:05.392594 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:05.392595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:05.392597 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:05.392639 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.392641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.393773 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.393774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.394709 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.394712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.395397 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.395397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.395483 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.395510 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 20:31:05.395608 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 20:31:05.395685 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":2,"thread_start_us":45,"threads_started":1}
I20260619 20:31:05.395768 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.392076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.21.151.3:39467)
0619 20:31:05.392239 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.392243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.392300 (+ 57us) server_negotiation.cc:407] Connection header received
0619 20:31:05.392371 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.392373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.392449 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.392526 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:05.392711 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.393670 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.394799 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.395294 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.395394 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.395453 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.395658 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.395699 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260619 20:31:05.395983 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.391520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:44009 (local address 127.0.0.1:54944)
0619 20:31:05.391639 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:05.391676 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:05.391735 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:05.392222 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:05.392223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:05.392225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:05.392278 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.392280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.393412 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.393413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.394766 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.394768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.395724 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.395725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.395789 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.395792 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:05.395822 (+ 30us) client_negotiation.cc:240] Negotiation successful
0619 20:31:05.395901 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":88,"thread_start_us":59,"threads_started":1}
I20260619 20:31:05.396147 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.391551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:39719 (local address 127.0.0.1:50002)
0619 20:31:05.391841 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:05.391852 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:05.391907 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:05.392091 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:05.392094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:05.392096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:05.392171 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.392173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.393484 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.393485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.394757 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.394760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.395778 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.395779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.395862 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.395866 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:31:05.395911 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 20:31:05.395942 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":30,"thread_start_us":66,"threads_started":1}
I20260619 20:31:05.396147 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.391664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.21.151.1:39719)
0619 20:31:05.391897 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.391903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.391914 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:31:05.391979 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.391981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.392002 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.392086 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:05.392261 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.393379 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.394861 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.395602 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.395694 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.395751 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.396025 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.396073 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260619 20:31:05.396322 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.391882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.21.151.2:44009)
0619 20:31:05.392039 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.392044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.392055 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:31:05.392095 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.392097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.392113 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.392365 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:05.392463 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.393301 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.394869 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.395597 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.396105 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.396170 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.396197 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.396242 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260619 20:31:05.396958 8034 tablet_service.cc:1467] Processing CreateTablet for tablet 6307cc3cf76f46ba857617ed2b4c6915 (DEFAULT_TABLE table=test-table [id=74bb8aa10ccf4ca1b225bd902d8eb4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:05.397047 7900 tablet_service.cc:1467] Processing CreateTablet for tablet 6307cc3cf76f46ba857617ed2b4c6915 (DEFAULT_TABLE table=test-table [id=74bb8aa10ccf4ca1b225bd902d8eb4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:05.397315 7900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6307cc3cf76f46ba857617ed2b4c6915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:05.397315 8034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6307cc3cf76f46ba857617ed2b4c6915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:05.400660 8248 tablet_bootstrap.cc:492] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9: Bootstrap starting.
I20260619 20:31:05.401001 8168 tablet_service.cc:1467] Processing CreateTablet for tablet 6307cc3cf76f46ba857617ed2b4c6915 (DEFAULT_TABLE table=test-table [id=74bb8aa10ccf4ca1b225bd902d8eb4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:05.401304 8168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6307cc3cf76f46ba857617ed2b4c6915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:05.401360 8248 tablet_bootstrap.cc:654] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:05.401849 8248 log.cc:824] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:05.402175 8249 tablet_bootstrap.cc:492] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: Bootstrap starting.
I20260619 20:31:05.402619 8248 tablet_bootstrap.cc:492] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9: No bootstrap required, opened a new log
I20260619 20:31:05.402694 8248 ts_tablet_manager.cc:1397] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:31:05.402735 8249 tablet_bootstrap.cc:654] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:05.403352 8249 log.cc:824] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:05.404474 8252 tablet_bootstrap.cc:492] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f: Bootstrap starting.
I20260619 20:31:05.404786 8248 raft_consensus.cc:348] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 } } peers { permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 } } peers { permanent_uuid: "333d29cb808547529acce01628b4436f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39467 } }
I20260619 20:31:05.404922 8248 raft_consensus.cc:374] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:05.404944 8248 raft_consensus.cc:729] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d61ccfd39c4e529c18edf3f75b3fd9, State: Initialized, Role: FOLLOWER
I20260619 20:31:05.405025 8252 tablet_bootstrap.cc:654] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:05.405079 8248 consensus_queue.cc:260] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 } } peers { permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 } } peers { permanent_uuid: "333d29cb808547529acce01628b4436f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39467 } }
I20260619 20:31:05.405400 7966 heartbeater.cc:503] Master 127.21.151.62:38413 was elected leader, sending a full tablet report...
I20260619 20:31:05.405385 8248 ts_tablet_manager.cc:1428] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:31:05.405578 8252 log.cc:824] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:05.405929 8249 tablet_bootstrap.cc:492] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: No bootstrap required, opened a new log
I20260619 20:31:05.405997 8249 ts_tablet_manager.cc:1397] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 20:31:05.406284 8252 tablet_bootstrap.cc:492] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f: No bootstrap required, opened a new log
I20260619 20:31:05.406338 8252 ts_tablet_manager.cc:1397] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:31:05.408048 8249 raft_consensus.cc:348] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 } } peers { permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 } } peers { permanent_uuid: "333d29cb808547529acce01628b4436f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39467 } }
I20260619 20:31:05.408242 8249 raft_consensus.cc:374] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:05.408279 8249 raft_consensus.cc:729] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb07ead79c4f4b49a11907a2c7609c80, State: Initialized, Role: FOLLOWER
I20260619 20:31:05.408185 8252 raft_consensus.cc:348] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 } } peers { permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 } } peers { permanent_uuid: "333d29cb808547529acce01628b4436f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39467 } }
I20260619 20:31:05.408329 8252 raft_consensus.cc:374] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:05.408368 8252 raft_consensus.cc:729] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333d29cb808547529acce01628b4436f, State: Initialized, Role: FOLLOWER
I20260619 20:31:05.408398 8249 consensus_queue.cc:260] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 } } peers { permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 } } peers { permanent_uuid: "333d29cb808547529acce01628b4436f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39467 } }
I20260619 20:31:05.408469 8252 consensus_queue.cc:260] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 } } peers { permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 } } peers { permanent_uuid: "333d29cb808547529acce01628b4436f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39467 } }
I20260619 20:31:05.408716 8252 ts_tablet_manager.cc:1428] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 20:31:05.408700 8100 heartbeater.cc:503] Master 127.21.151.62:38413 was elected leader, sending a full tablet report...
I20260619 20:31:05.408735 8234 heartbeater.cc:503] Master 127.21.151.62:38413 was elected leader, sending a full tablet report...
I20260619 20:31:05.408684 8249 ts_tablet_manager.cc:1428] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260619 20:31:05.447806 8101 tablet.cc:2369] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:31:05.536737 7967 tablet.cc:2369] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:31:05.613219 8235 tablet.cc:2369] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:31:05.616325 8256 raft_consensus.cc:482] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:31:05.616432 8256 raft_consensus.cc:504] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 } } peers { permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 } } peers { permanent_uuid: "333d29cb808547529acce01628b4436f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39467 } }
I20260619 20:31:05.616777 8256 leader_election.cc:288] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb07ead79c4f4b49a11907a2c7609c80 (127.21.151.2:44009), 48d61ccfd39c4e529c18edf3f75b3fd9 (127.21.151.1:39719)
I20260619 20:31:05.619911 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.616988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:44009 (local address 127.21.151.3:48103)
0619 20:31:05.617059 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:05.617070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:05.617118 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:05.617308 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:05.617309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:05.617310 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:05.617378 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.617380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.618131 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.618132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.619005 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.619007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.619607 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.619608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.619716 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.619720 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:31:05.619770 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:31:05.619800 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":35}
I20260619 20:31:05.619930 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.617058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:48103 (local address 127.21.151.2:44009)
0619 20:31:05.617103 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.617105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.617116 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:31:05.617193 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.617194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.617207 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.617297 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:05.617458 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.618039 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.619100 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.619538 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.619592 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.619599 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.619800 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.619849 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:31:05.619961 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.617038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:36579 (local address 127.21.151.1:39719)
0619 20:31:05.617080 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.617082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.617201 (+ 119us) server_negotiation.cc:407] Connection header received
0619 20:31:05.617299 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.617300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.617312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.617363 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:05.617509 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.618088 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.619172 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.619590 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.619686 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.619696 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.619846 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.619887 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:31:05.619983 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.616994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:39719 (local address 127.21.151.3:36579)
0619 20:31:05.617145 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:05.617154 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:05.617223 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:05.617369 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:05.617370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:05.617372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:05.617447 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.617449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.618173 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.618174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.619036 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:05.619067 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.619686 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.619687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:05.619748 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.619752 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:31:05.619833 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 20:31:05.619866 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":10,"thread_start_us":109,"threads_started":1}
I20260619 20:31:05.620316 8054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6307cc3cf76f46ba857617ed2b4c6915" candidate_uuid: "333d29cb808547529acce01628b4436f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb07ead79c4f4b49a11907a2c7609c80" is_pre_election: true
I20260619 20:31:05.620306 7920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6307cc3cf76f46ba857617ed2b4c6915" candidate_uuid: "333d29cb808547529acce01628b4436f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" is_pre_election: true
I20260619 20:31:05.620476 8054 raft_consensus.cc:2454] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333d29cb808547529acce01628b4436f in term 0.
I20260619 20:31:05.620476 7920 raft_consensus.cc:2454] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333d29cb808547529acce01628b4436f in term 0.
I20260619 20:31:05.620712 8124 leader_election.cc:302] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333d29cb808547529acce01628b4436f, eb07ead79c4f4b49a11907a2c7609c80; no voters:
I20260619 20:31:05.620883 8256 raft_consensus.cc:2781] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:31:05.620934 8256 raft_consensus.cc:482] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:31:05.620949 8256 raft_consensus.cc:3037] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:05.621594 8256 raft_consensus.cc:504] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 } } peers { permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 } } peers { permanent_uuid: "333d29cb808547529acce01628b4436f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39467 } }
I20260619 20:31:05.621742 8256 leader_election.cc:288] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [CANDIDATE]: Term 1 election: Requested vote from peers eb07ead79c4f4b49a11907a2c7609c80 (127.21.151.2:44009), 48d61ccfd39c4e529c18edf3f75b3fd9 (127.21.151.1:39719)
I20260619 20:31:05.621886 8054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6307cc3cf76f46ba857617ed2b4c6915" candidate_uuid: "333d29cb808547529acce01628b4436f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb07ead79c4f4b49a11907a2c7609c80"
I20260619 20:31:05.621912 7920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6307cc3cf76f46ba857617ed2b4c6915" candidate_uuid: "333d29cb808547529acce01628b4436f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9"
I20260619 20:31:05.621968 8054 raft_consensus.cc:3037] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:05.622004 7920 raft_consensus.cc:3037] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:05.622658 7920 raft_consensus.cc:2454] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333d29cb808547529acce01628b4436f in term 1.
I20260619 20:31:05.622674 8054 raft_consensus.cc:2454] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333d29cb808547529acce01628b4436f in term 1.
I20260619 20:31:05.622849 8124 leader_election.cc:302] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333d29cb808547529acce01628b4436f, eb07ead79c4f4b49a11907a2c7609c80; no voters:
I20260619 20:31:05.622943 8256 raft_consensus.cc:2781] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:05.623145 8256 raft_consensus.cc:686] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 1 LEADER]: Becoming Leader. State: Replica: 333d29cb808547529acce01628b4436f, State: Running, Role: LEADER
I20260619 20:31:05.623238 8256 consensus_queue.cc:237] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 } } peers { permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 } } peers { permanent_uuid: "333d29cb808547529acce01628b4436f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39467 } }
I20260619 20:31:05.624073 7774 catalog_manager.cc:5495] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f reported cstate change: term changed from 0 to 1, leader changed from <none> to 333d29cb808547529acce01628b4436f (127.21.151.3). New cstate: current_term: 1 leader_uuid: "333d29cb808547529acce01628b4436f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333d29cb808547529acce01628b4436f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39467 } health_report { overall_health: HEALTHY } } }
I20260619 20:31:05.678593 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.676101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.21.151.3:39467)
0619 20:31:05.676162 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.676164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.676171 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:31:05.676206 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.676207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.676224 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.676303 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:05.676449 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.677022 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.677779 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.678006 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.678082 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.678221 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:05.678319 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:05.678428 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.678477 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.678488 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:31:05.680603 7920 raft_consensus.cc:1261] T 6307cc3cf76f46ba857617ed2b4c6915 P 48d61ccfd39c4e529c18edf3f75b3fd9 [term 1 FOLLOWER]: Refusing update from remote peer 333d29cb808547529acce01628b4436f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:05.680606 8054 raft_consensus.cc:1261] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80 [term 1 FOLLOWER]: Refusing update from remote peer 333d29cb808547529acce01628b4436f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:05.680977 8256 consensus_queue.cc:1035] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb07ead79c4f4b49a11907a2c7609c80" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 44009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:05.681147 8258 consensus_queue.cc:1035] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d61ccfd39c4e529c18edf3f75b3fd9" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 39719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:05.683879 8262 mvcc.cc:205] Tried to move back new op lower bound from 7298666765024874496 to 7298666764793442304. Current Snapshot: MvccSnapshot[applied={T|T < 7298666765024874496}]
I20260619 20:31:05.684288 8261 mvcc.cc:205] Tried to move back new op lower bound from 7298666765024874496 to 7298666764793442304. Current Snapshot: MvccSnapshot[applied={T|T < 7298666765024874496 or (T in {7298666765024874496})}]
I20260619 20:31:05.685837 7774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:05.685942 7774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:05.686681 7774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69b5ec2894f7402b9637a17d04732754: Sending DeleteTablet for 3 replicas of tablet 6307cc3cf76f46ba857617ed2b4c6915
I20260619 20:31:05.687043 8034 tablet_service.cc:1514] Processing DeleteTablet for tablet 6307cc3cf76f46ba857617ed2b4c6915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:05 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:54944
I20260619 20:31:05.687052 7900 tablet_service.cc:1514] Processing DeleteTablet for tablet 6307cc3cf76f46ba857617ed2b4c6915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:05 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:50002
I20260619 20:31:05.687316 8270 tablet_replica.cc:330] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: stopping tablet replica
I20260619 20:31:05.687306 8168 tablet_service.cc:1514] Processing DeleteTablet for tablet 6307cc3cf76f46ba857617ed2b4c6915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:05 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:36826
I20260619 20:31:05.687410 8270 raft_consensus.cc:2229] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:31:05.687520 8270 raft_consensus.cc:2258] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:05.687543 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 7837
I20260619 20:31:05.690197 8270 ts_tablet_manager.cc:1905] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:05.692178 8270 ts_tablet_manager.cc:1918] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:05.692255 8270 log.cc:1196] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/6307cc3cf76f46ba857617ed2b4c6915
I20260619 20:31:05.692530 8270 ts_tablet_manager.cc:1939] T 6307cc3cf76f46ba857617ed2b4c6915 P eb07ead79c4f4b49a11907a2c7609c80: Deleting consensus metadata
I20260619 20:31:05.692953 7761 catalog_manager.cc:4841] TS eb07ead79c4f4b49a11907a2c7609c80 (127.21.151.2:44009): tablet 6307cc3cf76f46ba857617ed2b4c6915 (table test-table [id=74bb8aa10ccf4ca1b225bd902d8eb4fe]) successfully deleted
W20260619 20:31:05.693320 7760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:39719 (error 108)
I20260619 20:31:05.693857 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 7971
I20260619 20:31:05.694514 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.693936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:39719 (local address 127.0.0.1:50008)
0619 20:31:05.694431 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:05.694476 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:39719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461}
W20260619 20:31:05.695731 7760 catalog_manager.cc:4584] TS 48d61ccfd39c4e529c18edf3f75b3fd9 (127.21.151.1:39719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6307cc3cf76f46ba857617ed2b4c6915: Network error: Client connection negotiation failed: client connection to 127.21.151.1:39719: connect: Connection refused (error 111)
I20260619 20:31:05.696028 8272 tablet_replica.cc:330] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f: stopping tablet replica
I20260619 20:31:05.696147 8272 raft_consensus.cc:2229] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:31:05.696239 8272 raft_consensus.cc:2258] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:05.700111 8272 ts_tablet_manager.cc:1905] T 6307cc3cf76f46ba857617ed2b4c6915 P 333d29cb808547529acce01628b4436f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:05.700513 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 8105
I20260619 20:31:05.705837 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 7742
2026-06-19T20:31:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2107 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T20:31:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:05Z Disabled control of system clock
I20260619 20:31:05.733042 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:33133
--webserver_interface=127.21.151.62
--webserver_port=0
--builtin_ntp_servers=127.21.151.20:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:33133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:31:05.812420 8277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:05.812618 8277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:05.812635 8277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:05.813759 8277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:05.813800 8277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:05.813829 8277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:05.813841 8277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:05.813853 8277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:05.815193 8277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:33133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:33133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:05.815440 8277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:05.815738 8277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:05.820587 8283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:05.820609 8282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:05.820873 8277 server_base.cc:1034] running on GCE node
W20260619 20:31:05.820621 8285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:05.821193 8277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:05.821570 8277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:05.822726 8277 hybrid_clock.cc:648] HybridClock initialized: now 1781901065822712 us; error 35 us; skew 500 ppm
I20260619 20:31:05.823930 8277 webserver.cc:460] Webserver started at http://127.21.151.62:37471/ using document root <none> and password file <none>
I20260619 20:31:05.824160 8277 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:05.824229 8277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:05.824386 8277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:05.825464 8277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "f8c4497c29c548ab901a6d8114c8fa6b"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "90019dc4acc08a40fcff3591a295847c"
server_key_iv: "3b9c13e13ce05dc938b42ecf3ee34fa3"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.825806 8277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "f8c4497c29c548ab901a6d8114c8fa6b"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "90019dc4acc08a40fcff3591a295847c"
server_key_iv: "3b9c13e13ce05dc938b42ecf3ee34fa3"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.827015 8277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:31:05.827742 8291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.827924 8277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:05.828017 8277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "f8c4497c29c548ab901a6d8114c8fa6b"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "90019dc4acc08a40fcff3591a295847c"
server_key_iv: "3b9c13e13ce05dc938b42ecf3ee34fa3"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.828102 8277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:05.839792 8277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:05.840080 8277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:05.840233 8277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:05.843845 8277 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:33133
I20260619 20:31:05.843899 8343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:33133 every 8 connection(s)
I20260619 20:31:05.844256 8277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:05.845227 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:05.847076 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 8277
I20260619 20:31:05.847188 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:05.847376 22108 external_mini_cluster.cc:1428] Setting key ba2bb7ee86eaa06ad6d51fbb88bfae56
I20260619 20:31:05.848026 8344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b: Bootstrap starting.
I20260619 20:31:05.848606 8344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:05.848906 8344 log.cc:824] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:05.849620 8344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b: No bootstrap required, opened a new log
I20260619 20:31:05.851846 8344 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c4497c29c548ab901a6d8114c8fa6b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33133 } }
I20260619 20:31:05.852025 8344 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:05.852073 8344 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c4497c29c548ab901a6d8114c8fa6b, State: Initialized, Role: FOLLOWER
I20260619 20:31:05.852201 8344 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c4497c29c548ab901a6d8114c8fa6b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33133 } }
I20260619 20:31:05.852313 8344 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:05.852346 8344 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:05.852384 8344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:05.853129 8344 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c4497c29c548ab901a6d8114c8fa6b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33133 } }
I20260619 20:31:05.853236 8344 leader_election.cc:302] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8c4497c29c548ab901a6d8114c8fa6b; no voters:
I20260619 20:31:05.853379 8344 leader_election.cc:288] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:05.853417 8349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:05.853477 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:05.847933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.21.151.62:33133)
0619 20:31:05.848190 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:05.848200 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:05.848230 (+ 30us) server_negotiation.cc:407] Connection header received
0619 20:31:05.848634 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:05.848661 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:05.848740 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:05.848889 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:05.849096 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.850435 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.851200 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:05.851913 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:05.852028 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:05.852729 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:05.852759 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:05.852760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:05.852977 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:05.853020 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:05.853034 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:05.853104 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:05.853161 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 20:31:05.853237 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":122,"threads_started":1}
I20260619 20:31:05.853564 8349 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [term 1 LEADER]: Becoming Leader. State: Replica: f8c4497c29c548ab901a6d8114c8fa6b, State: Running, Role: LEADER
I20260619 20:31:05.853615 8344 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:05.853647 8349 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c4497c29c548ab901a6d8114c8fa6b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33133 } }
I20260619 20:31:05.853996 8349 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8c4497c29c548ab901a6d8114c8fa6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c4497c29c548ab901a6d8114c8fa6b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33133 } } }
I20260619 20:31:05.854070 8349 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:05.854688 8350 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8c4497c29c548ab901a6d8114c8fa6b. Latest consensus state: current_term: 1 leader_uuid: "f8c4497c29c548ab901a6d8114c8fa6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c4497c29c548ab901a6d8114c8fa6b" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33133 } } }
I20260619 20:31:05.854765 8350 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:05.854812 8356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:05.855347 8356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:05.856858 8356 catalog_manager.cc:1347] Generated new cluster ID: 392defd8891749dea5ebe46aed62154b
I20260619 20:31:05.856915 8356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:05.880061 8356 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:05.880770 8356 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:05.885298 8356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b: Generated new TSK 0
I20260619 20:31:05.885470 8356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:05.887913 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33133
--builtin_ntp_servers=127.21.151.20:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:31:05.970554 8368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:05.970744 8368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:05.970765 8368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:05.971951 8368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:05.971999 8368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:05.972023 8368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:31:05.973383 8368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:05.973611 8368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:05.973927 8368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:05.979838 8374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:05.979861 8376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:05.979964 8368 server_base.cc:1034] running on GCE node
W20260619 20:31:05.980083 8373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:05.980273 8368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:05.980527 8368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:05.981689 8368 hybrid_clock.cc:648] HybridClock initialized: now 1781901065981675 us; error 28 us; skew 500 ppm
I20260619 20:31:05.982998 8368 webserver.cc:460] Webserver started at http://127.21.151.1:38905/ using document root <none> and password file <none>
I20260619 20:31:05.983240 8368 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:05.983327 8368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:05.983481 8368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:05.984648 8368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "6341ffc93ec94a95861dce16480fc6b2"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "778a356a56118a30d0ff46708d513b32"
server_key_iv: "ad87705ed6af7e01acaf358c386a29fa"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.985005 8368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "6341ffc93ec94a95861dce16480fc6b2"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "778a356a56118a30d0ff46708d513b32"
server_key_iv: "ad87705ed6af7e01acaf358c386a29fa"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.986351 8368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 20:31:05.987047 8382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:05.987219 8368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:05.987311 8368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "6341ffc93ec94a95861dce16480fc6b2"
format_stamp: "Formatted at 2026-06-19 20:31:05 on dist-test-slave-f9rk"
server_key: "778a356a56118a30d0ff46708d513b32"
server_key_iv: "ad87705ed6af7e01acaf358c386a29fa"
server_key_version: "encryptionkey@0"
I20260619 20:31:05.987391 8368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:06.007901 8368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:06.008252 8368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:06.008436 8368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:06.008673 8368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:06.009019 8368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:06.009079 8368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:06.009120 8368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:06.009148 8368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:06.014716 8368 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:46755
I20260619 20:31:06.014803 8495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:46755 every 8 connection(s)
I20260619 20:31:06.015120 8368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:31:06.021310 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.015886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:38991 (local address 127.21.151.62:33133)
0619 20:31:06.015943 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:06.015945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:06.016276 (+ 331us) server_negotiation.cc:407] Connection header received
0619 20:31:06.016867 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:06.016869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:06.016895 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:06.016949 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:06.017863 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.018499 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.019699 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.020031 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.020115 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.021058 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:06.021060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:06.021060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:06.021065 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:06.021080 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:06.021085 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:06.021128 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:06.021204 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 20:31:06.021230 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:31:06.021543 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.015900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33133 (local address 127.21.151.1:38991)
0619 20:31:06.016125 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:06.016171 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:06.016758 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:06.017031 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:06.017035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:06.017045 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:06.017746 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.017754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.018595 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.018596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.019502 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.019522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.020128 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.020129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.020274 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.020902 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:31:06.020902 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:06.020929 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:31:06.020951 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:06.021139 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:06.021144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:06.021147 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:06.021210 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 20:31:06.021311 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260619 20:31:06.022053 8496 heartbeater.cc:347] Connected to a master server at 127.21.151.62:33133
I20260619 20:31:06.022168 8496 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:06.022401 8496 heartbeater.cc:511] Master 127.21.151.62:33133 requested a full tablet report, sending...
I20260619 20:31:06.022951 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 8368
I20260619 20:31:06.022992 8308 ts_manager.cc:194] Registered new tserver with Master: 6341ffc93ec94a95861dce16480fc6b2 (127.21.151.1:46755)
I20260619 20:31:06.023066 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:31:06.023298 22108 external_mini_cluster.cc:1428] Setting key 5da01f407c3ba01afad56c5aa77b1118
I20260619 20:31:06.024195 8308 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.1:38991
I20260619 20:31:06.024776 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33133
--builtin_ntp_servers=127.21.151.20:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:31:06.108022 8499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:06.108235 8499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:06.108258 8499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:06.109417 8499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:06.109454 8499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:06.109479 8499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:31:06.110868 8499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:06.111064 8499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:06.111325 8499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:06.116199 8505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:06.116206 8504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:06.116209 8507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:06.116459 8499 server_base.cc:1034] running on GCE node
I20260619 20:31:06.116739 8499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:06.117084 8499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:06.118252 8499 hybrid_clock.cc:648] HybridClock initialized: now 1781901066118226 us; error 39 us; skew 500 ppm
I20260619 20:31:06.119474 8499 webserver.cc:460] Webserver started at http://127.21.151.2:40673/ using document root <none> and password file <none>
I20260619 20:31:06.119786 8499 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:06.119859 8499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:06.120033 8499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:06.121168 8499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "614dc2c2391e4a81b561cf65e5fb5c19"
format_stamp: "Formatted at 2026-06-19 20:31:06 on dist-test-slave-f9rk"
server_key: "7ab0c54262c0ff17391c353b7ceff77e"
server_key_iv: "bc7728af7bfda8da1aacf6c98e2a9cd6"
server_key_version: "encryptionkey@0"
I20260619 20:31:06.121547 8499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "614dc2c2391e4a81b561cf65e5fb5c19"
format_stamp: "Formatted at 2026-06-19 20:31:06 on dist-test-slave-f9rk"
server_key: "7ab0c54262c0ff17391c353b7ceff77e"
server_key_iv: "bc7728af7bfda8da1aacf6c98e2a9cd6"
server_key_version: "encryptionkey@0"
I20260619 20:31:06.122874 8499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:31:06.123805 8513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:06.123998 8499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:31:06.124097 8499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "614dc2c2391e4a81b561cf65e5fb5c19"
format_stamp: "Formatted at 2026-06-19 20:31:06 on dist-test-slave-f9rk"
server_key: "7ab0c54262c0ff17391c353b7ceff77e"
server_key_iv: "bc7728af7bfda8da1aacf6c98e2a9cd6"
server_key_version: "encryptionkey@0"
I20260619 20:31:06.124186 8499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:06.141296 8499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:06.141644 8499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:06.141803 8499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:06.142048 8499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:06.142409 8499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:06.142472 8499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:06.142515 8499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:06.142540 8499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:06.148461 8499 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:42121
I20260619 20:31:06.148490 8626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:42121 every 8 connection(s)
I20260619 20:31:06.148828 8499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:31:06.150681 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 8499
I20260619 20:31:06.150804 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:31:06.151062 22108 external_mini_cluster.cc:1428] Setting key 509aef6848ead53d13361f1156c5dd54
I20260619 20:31:06.152122 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33133
--builtin_ntp_servers=127.21.151.20:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 20:31:06.155269 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.149558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:40889 (local address 127.21.151.62:33133)
0619 20:31:06.149625 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:06.149627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:06.149858 (+ 231us) server_negotiation.cc:407] Connection header received
0619 20:31:06.150322 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:06.150323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:06.150333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:06.150383 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:06.151315 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.152028 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.153576 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.153909 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.154009 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.154941 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:06.154943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:06.154943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:06.154948 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:06.154966 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:06.154970 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:06.155025 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:06.155124 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 20:31:06.155151 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:31:06.155378 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.149547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33133 (local address 127.21.151.2:40889)
0619 20:31:06.149724 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:06.149760 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:06.150238 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:06.150452 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:06.150455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:06.150464 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:06.151215 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.151222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.152124 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.152125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.153475 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.153478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.154018 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.154019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.154143 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.154784 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:31:06.154784 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:06.154813 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:31:06.154835 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:06.155042 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:06.155047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:06.155050 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:06.155107 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 20:31:06.155188 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260619 20:31:06.155905 8627 heartbeater.cc:347] Connected to a master server at 127.21.151.62:33133
I20260619 20:31:06.156014 8627 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:06.156203 8627 heartbeater.cc:511] Master 127.21.151.62:33133 requested a full tablet report, sending...
I20260619 20:31:06.156620 8308 ts_manager.cc:194] Registered new tserver with Master: 614dc2c2391e4a81b561cf65e5fb5c19 (127.21.151.2:42121)
I20260619 20:31:06.157167 8308 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.2:40889
W20260619 20:31:06.234938 8630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:06.235132 8630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:06.235152 8630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:06.236269 8630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:06.236307 8630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:06.236331 8630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:31:06.237618 8630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:06.237792 8630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:06.238013 8630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:06.242698 8638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:06.242704 8636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:06.242715 8635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:06.242921 8630 server_base.cc:1034] running on GCE node
I20260619 20:31:06.243268 8630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:06.243541 8630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:06.244719 8630 hybrid_clock.cc:648] HybridClock initialized: now 1781901066244701 us; error 33 us; skew 500 ppm
I20260619 20:31:06.245889 8630 webserver.cc:460] Webserver started at http://127.21.151.3:45357/ using document root <none> and password file <none>
I20260619 20:31:06.246143 8630 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:06.246214 8630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:06.246353 8630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:06.247416 8630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "9b3a9e96c5834ba2900e2201ec557eed"
format_stamp: "Formatted at 2026-06-19 20:31:06 on dist-test-slave-f9rk"
server_key: "2531ff5c4cbe027ffc9f448524903b01"
server_key_iv: "bb82dccdb3b32e4086812f9466fce546"
server_key_version: "encryptionkey@0"
I20260619 20:31:06.247807 8630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "9b3a9e96c5834ba2900e2201ec557eed"
format_stamp: "Formatted at 2026-06-19 20:31:06 on dist-test-slave-f9rk"
server_key: "2531ff5c4cbe027ffc9f448524903b01"
server_key_iv: "bb82dccdb3b32e4086812f9466fce546"
server_key_version: "encryptionkey@0"
I20260619 20:31:06.249100 8630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:31:06.249826 8644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:06.250010 8630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:06.250109 8630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "9b3a9e96c5834ba2900e2201ec557eed"
format_stamp: "Formatted at 2026-06-19 20:31:06 on dist-test-slave-f9rk"
server_key: "2531ff5c4cbe027ffc9f448524903b01"
server_key_iv: "bb82dccdb3b32e4086812f9466fce546"
server_key_version: "encryptionkey@0"
I20260619 20:31:06.250197 8630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:06.266243 8630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:06.266568 8630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:06.266733 8630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:06.266983 8630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:06.267324 8630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:06.267379 8630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:06.267416 8630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:06.267445 8630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:06.273221 8630 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:36037
I20260619 20:31:06.273301 8759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:36037 every 8 connection(s)
I20260619 20:31:06.273645 8630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:31:06.278066 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 8630
I20260619 20:31:06.278192 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:31:06.278399 22108 external_mini_cluster.cc:1428] Setting key 0f1bd57666942855d6b56eaf0eba112b
I20260619 20:31:06.279090 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.274470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:47881 (local address 127.21.151.62:33133)
0619 20:31:06.274549 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:06.274552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:06.274761 (+ 209us) server_negotiation.cc:407] Connection header received
0619 20:31:06.275287 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:06.275289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:06.275300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:06.275354 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:06.276040 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.276555 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.277542 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.277753 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.277832 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.278846 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:06.278847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:06.278847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:06.278852 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:06.278865 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:06.278869 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:06.278927 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:06.278995 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 20:31:06.279016 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:31:06.279307 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.274439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33133 (local address 127.21.151.3:47881)
0619 20:31:06.274643 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:06.274675 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:06.275183 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:06.275428 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:06.275432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:06.275441 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:06.275949 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.275954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.276661 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.276662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.277451 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.277454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.277882 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.277883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.278063 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.278692 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:31:06.278692 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:06.278740 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:31:06.278768 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:06.278927 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:06.278931 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:06.278934 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:06.278998 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 20:31:06.279092 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260619 20:31:06.279810 8760 heartbeater.cc:347] Connected to a master server at 127.21.151.62:33133
I20260619 20:31:06.279898 8760 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:06.280040 8760 heartbeater.cc:511] Master 127.21.151.62:33133 requested a full tablet report, sending...
I20260619 20:31:06.280375 8308 ts_manager.cc:194] Registered new tserver with Master: 9b3a9e96c5834ba2900e2201ec557eed (127.21.151.3:36037)
I20260619 20:31:06.280761 8308 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.3:47881
I20260619 20:31:06.290524 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:31:06.295073 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.291936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.21.151.62:33133)
0619 20:31:06.292002 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:06.292004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:06.292153 (+ 149us) server_negotiation.cc:407] Connection header received
0619 20:31:06.292219 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:06.292219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:06.292230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:06.292285 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:06.292452 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.292942 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.293711 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.293919 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.293990 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.294736 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:06.294737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:06.294737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:06.294741 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:06.294755 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:06.294759 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:06.294902 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:06.294971 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 20:31:06.294982 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 20:31:06.296911 8308 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:31:06.298406 8308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:31:06.304745 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.300821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.21.151.1:46755)
0619 20:31:06.301026 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:06.301030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:06.301041 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:31:06.301086 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:06.301088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:06.301106 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:06.301181 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:06.301480 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.302628 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.303793 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.304343 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.304430 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.304528 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:06.304621 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 20:31:06.304659 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":99,"threads_started":1}
I20260619 20:31:06.304735 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.300586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:46755 (local address 127.0.0.1:35508)
0619 20:31:06.300772 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:06.300811 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:06.300881 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:06.301288 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:06.301291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:06.301294 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:06.301362 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.301364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.302728 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.302729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.303708 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.303711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.304451 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.304452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.304528 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.304542 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:31:06.304610 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 20:31:06.304659 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
I20260619 20:31:06.304934 8772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.300820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.21.151.3:36037)
0619 20:31:06.301220 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:06.301223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:06.301232 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:31:06.301269 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:06.301272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:06.301286 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:06.301354 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:06.301546 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.302246 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.303677 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.304395 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.304463 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.304545 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:06.304788 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 20:31:06.304830 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":81,"threads_started":1}
I20260619 20:31:06.305104 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.300604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:36037 (local address 127.0.0.1:41504)
0619 20:31:06.301052 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:06.301061 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:06.301106 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:06.301432 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:06.301433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:06.301435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:06.301472 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.301473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.302340 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.302341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.303562 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.303564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.304582 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.304583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.304646 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.304650 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:31:06.304703 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:31:06.304744 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":80,"threads_started":1}
I20260619 20:31:06.305207 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.300697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.21.151.2:42121)
0619 20:31:06.300893 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:06.300899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:06.301114 (+ 215us) server_negotiation.cc:407] Connection header received
0619 20:31:06.301167 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:06.301170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:06.301184 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:06.301282 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:06.301554 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.302636 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.303954 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.304664 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.304722 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.304796 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:06.305074 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 20:31:06.305116 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260619 20:31:06.305442 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.300588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:42121 (local address 127.0.0.1:57912)
0619 20:31:06.301012 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:06.301025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:06.301105 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:06.301416 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:06.301417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:06.301419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:06.301462 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.301464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.302741 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.302742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.303872 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.303874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.304827 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.304828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.304885 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.304888 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:06.304937 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 20:31:06.304977 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":30,"thread_start_us":59,"threads_started":1}
I20260619 20:31:06.305816 8692 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd2e866a2404c78b16379ba988ae374 (DEFAULT_TABLE table=test-table [id=23f3764840fa421f9dffd3e08e16129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:06.305887 8430 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd2e866a2404c78b16379ba988ae374 (DEFAULT_TABLE table=test-table [id=23f3764840fa421f9dffd3e08e16129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:06.306164 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd2e866a2404c78b16379ba988ae374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:06.306273 8430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd2e866a2404c78b16379ba988ae374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:06.308811 8774 tablet_bootstrap.cc:492] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: Bootstrap starting.
I20260619 20:31:06.308827 8775 tablet_bootstrap.cc:492] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2: Bootstrap starting.
I20260619 20:31:06.309496 8775 tablet_bootstrap.cc:654] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:06.309533 8774 tablet_bootstrap.cc:654] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:06.309540 8561 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd2e866a2404c78b16379ba988ae374 (DEFAULT_TABLE table=test-table [id=23f3764840fa421f9dffd3e08e16129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:06.309824 8775 log.cc:824] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:06.309850 8774 log.cc:824] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:06.309873 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd2e866a2404c78b16379ba988ae374. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:06.310652 8775 tablet_bootstrap.cc:492] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2: No bootstrap required, opened a new log
I20260619 20:31:06.310726 8775 ts_tablet_manager.cc:1397] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:31:06.310822 8774 tablet_bootstrap.cc:492] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: No bootstrap required, opened a new log
I20260619 20:31:06.310912 8774 ts_tablet_manager.cc:1397] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:31:06.312898 8774 raft_consensus.cc:348] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6341ffc93ec94a95861dce16480fc6b2" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46755 } } peers { permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 } } peers { permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 } }
I20260619 20:31:06.313058 8774 raft_consensus.cc:374] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:06.313102 8774 raft_consensus.cc:729] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3a9e96c5834ba2900e2201ec557eed, State: Initialized, Role: FOLLOWER
I20260619 20:31:06.313235 8774 consensus_queue.cc:260] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6341ffc93ec94a95861dce16480fc6b2" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46755 } } peers { permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 } } peers { permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 } }
I20260619 20:31:06.313380 8778 tablet_bootstrap.cc:492] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: Bootstrap starting.
I20260619 20:31:06.313308 8775 raft_consensus.cc:348] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6341ffc93ec94a95861dce16480fc6b2" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46755 } } peers { permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 } } peers { permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 } }
I20260619 20:31:06.313457 8775 raft_consensus.cc:374] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:06.313496 8774 ts_tablet_manager.cc:1428] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:31:06.313503 8775 raft_consensus.cc:729] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6341ffc93ec94a95861dce16480fc6b2, State: Initialized, Role: FOLLOWER
I20260619 20:31:06.313620 8760 heartbeater.cc:503] Master 127.21.151.62:33133 was elected leader, sending a full tablet report...
I20260619 20:31:06.313617 8775 consensus_queue.cc:260] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6341ffc93ec94a95861dce16480fc6b2" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46755 } } peers { permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 } } peers { permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 } }
I20260619 20:31:06.313877 8775 ts_tablet_manager.cc:1428] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 20:31:06.313908 8496 heartbeater.cc:503] Master 127.21.151.62:33133 was elected leader, sending a full tablet report...
I20260619 20:31:06.314462 8778 tablet_bootstrap.cc:654] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:06.314774 8778 log.cc:824] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:06.315384 8778 tablet_bootstrap.cc:492] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: No bootstrap required, opened a new log
I20260619 20:31:06.315433 8778 ts_tablet_manager.cc:1397] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:31:06.317134 8778 raft_consensus.cc:348] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6341ffc93ec94a95861dce16480fc6b2" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46755 } } peers { permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 } } peers { permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 } }
I20260619 20:31:06.317250 8778 raft_consensus.cc:374] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:06.317270 8778 raft_consensus.cc:729] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 614dc2c2391e4a81b561cf65e5fb5c19, State: Initialized, Role: FOLLOWER
I20260619 20:31:06.317346 8778 consensus_queue.cc:260] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6341ffc93ec94a95861dce16480fc6b2" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46755 } } peers { permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 } } peers { permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 } }
I20260619 20:31:06.317520 8778 ts_tablet_manager.cc:1428] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:31:06.317588 8627 heartbeater.cc:503] Master 127.21.151.62:33133 was elected leader, sending a full tablet report...
W20260619 20:31:06.399355 8628 tablet.cc:2369] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:31:06.485499 8780 raft_consensus.cc:482] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:31:06.485653 8780 raft_consensus.cc:504] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6341ffc93ec94a95861dce16480fc6b2" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46755 } } peers { permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 } } peers { permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 } }
I20260619 20:31:06.486056 8780 leader_election.cc:288] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 614dc2c2391e4a81b561cf65e5fb5c19 (127.21.151.2:42121), 9b3a9e96c5834ba2900e2201ec557eed (127.21.151.3:36037)
I20260619 20:31:06.489579 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.486326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:57479 (local address 127.21.151.2:42121)
0619 20:31:06.486381 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:06.486383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:06.486521 (+ 138us) server_negotiation.cc:407] Connection header received
0619 20:31:06.486574 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:06.486575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:06.486585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:06.486643 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:06.486824 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.487324 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.488843 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.489207 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.489268 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.489275 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:06.489436 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 20:31:06.489498 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:31:06.489665 8772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.486348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:52509 (local address 127.21.151.3:36037)
0619 20:31:06.486437 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:06.486439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:06.486449 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:31:06.486482 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:06.486483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:06.486493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:06.486551 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:06.486732 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.487426 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.488889 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.489308 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.489379 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.489387 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:06.489551 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 20:31:06.489595 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 20:31:06.489696 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.486244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:36037 (local address 127.21.151.1:52509)
0619 20:31:06.486338 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:06.486351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:06.486397 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:06.486573 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:06.486574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:06.486576 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:06.486650 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.486652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.487507 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.487508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.488763 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.488766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.489408 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.489409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.489485 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.489488 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:06.489542 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 20:31:06.489603 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":47}
I20260619 20:31:06.489609 8783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.486269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:42121 (local address 127.21.151.1:57479)
0619 20:31:06.486433 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:06.486446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:06.486512 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:06.486660 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:06.486661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:06.486662 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:06.486744 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.486746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.487411 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.487412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.488766 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:06.488768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.489291 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.489292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:06.489373 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.489377 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:31:06.489433 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 20:31:06.489523 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":41,"thread_start_us":99,"threads_started":1}
I20260619 20:31:06.490131 8713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd2e866a2404c78b16379ba988ae374" candidate_uuid: "6341ffc93ec94a95861dce16480fc6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3a9e96c5834ba2900e2201ec557eed" is_pre_election: true
I20260619 20:31:06.490115 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd2e866a2404c78b16379ba988ae374" candidate_uuid: "6341ffc93ec94a95861dce16480fc6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" is_pre_election: true
I20260619 20:31:06.490279 8581 raft_consensus.cc:2454] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6341ffc93ec94a95861dce16480fc6b2 in term 0.
I20260619 20:31:06.490279 8713 raft_consensus.cc:2454] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6341ffc93ec94a95861dce16480fc6b2 in term 0.
I20260619 20:31:06.490516 8385 leader_election.cc:302] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6341ffc93ec94a95861dce16480fc6b2, 9b3a9e96c5834ba2900e2201ec557eed; no voters:
I20260619 20:31:06.490634 8780 raft_consensus.cc:2781] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:31:06.490736 8780 raft_consensus.cc:482] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:31:06.490772 8780 raft_consensus.cc:3037] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:06.491525 8780 raft_consensus.cc:504] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6341ffc93ec94a95861dce16480fc6b2" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46755 } } peers { permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 } } peers { permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 } }
I20260619 20:31:06.491698 8780 leader_election.cc:288] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [CANDIDATE]: Term 1 election: Requested vote from peers 614dc2c2391e4a81b561cf65e5fb5c19 (127.21.151.2:42121), 9b3a9e96c5834ba2900e2201ec557eed (127.21.151.3:36037)
I20260619 20:31:06.491827 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd2e866a2404c78b16379ba988ae374" candidate_uuid: "6341ffc93ec94a95861dce16480fc6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "614dc2c2391e4a81b561cf65e5fb5c19"
I20260619 20:31:06.491860 8713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd2e866a2404c78b16379ba988ae374" candidate_uuid: "6341ffc93ec94a95861dce16480fc6b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3a9e96c5834ba2900e2201ec557eed"
I20260619 20:31:06.491938 8581 raft_consensus.cc:3037] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:06.491973 8713 raft_consensus.cc:3037] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:06.492682 8581 raft_consensus.cc:2454] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6341ffc93ec94a95861dce16480fc6b2 in term 1.
I20260619 20:31:06.492702 8713 raft_consensus.cc:2454] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6341ffc93ec94a95861dce16480fc6b2 in term 1.
I20260619 20:31:06.492863 8383 leader_election.cc:302] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 614dc2c2391e4a81b561cf65e5fb5c19, 6341ffc93ec94a95861dce16480fc6b2; no voters:
I20260619 20:31:06.492956 8780 raft_consensus.cc:2781] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:06.493090 8780 raft_consensus.cc:686] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [term 1 LEADER]: Becoming Leader. State: Replica: 6341ffc93ec94a95861dce16480fc6b2, State: Running, Role: LEADER
I20260619 20:31:06.493207 8780 consensus_queue.cc:237] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6341ffc93ec94a95861dce16480fc6b2" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46755 } } peers { permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 } } peers { permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 } }
I20260619 20:31:06.494103 8308 catalog_manager.cc:5495] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6341ffc93ec94a95861dce16480fc6b2 (127.21.151.1). New cstate: current_term: 1 leader_uuid: "6341ffc93ec94a95861dce16480fc6b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6341ffc93ec94a95861dce16480fc6b2" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 46755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 } health_report { overall_health: UNKNOWN } } }
W20260619 20:31:06.515892 8497 tablet.cc:2369] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:31:06.524241 8761 tablet.cc:2369] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:31:06.533551 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.531393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.21.151.1:46755)
0619 20:31:06.531464 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:06.531466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:06.531474 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:31:06.531511 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:06.531512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:06.531530 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:06.531594 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:06.531733 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.532204 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.532927 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:06.533107 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:06.533157 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:06.533273 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:06.533355 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:06.533425 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:06.533467 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 20:31:06.533476 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 20:31:06.535290 8581 raft_consensus.cc:1261] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19 [term 1 FOLLOWER]: Refusing update from remote peer 6341ffc93ec94a95861dce16480fc6b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:06.535297 8713 raft_consensus.cc:1261] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed [term 1 FOLLOWER]: Refusing update from remote peer 6341ffc93ec94a95861dce16480fc6b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:06.535652 8784 consensus_queue.cc:1035] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "614dc2c2391e4a81b561cf65e5fb5c19" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:06.535784 8780 consensus_queue.cc:1035] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3a9e96c5834ba2900e2201ec557eed" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 36037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:06.540498 8308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:06.540606 8308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:06.541355 8308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8c4497c29c548ab901a6d8114c8fa6b: Sending DeleteTablet for 3 replicas of tablet 5cd2e866a2404c78b16379ba988ae374
I20260619 20:31:06.541662 8430 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd2e866a2404c78b16379ba988ae374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:06 UTC) from {username='slave'} at 127.0.0.1:35508
I20260619 20:31:06.541690 8561 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd2e866a2404c78b16379ba988ae374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:06 UTC) from {username='slave'} at 127.0.0.1:57912
I20260619 20:31:06.541795 8692 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd2e866a2404c78b16379ba988ae374 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:06 UTC) from {username='slave'} at 127.0.0.1:41504
I20260619 20:31:06.541987 8797 tablet_replica.cc:330] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: stopping tablet replica
I20260619 20:31:06.542017 8798 tablet_replica.cc:330] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: stopping tablet replica
I20260619 20:31:06.542093 8797 raft_consensus.cc:2229] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:31:06.542192 8797 raft_consensus.cc:2258] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:06.542241 8796 tablet_replica.cc:330] T 5cd2e866a2404c78b16379ba988ae374 P 6341ffc93ec94a95861dce16480fc6b2: stopping tablet replica
I20260619 20:31:06.542310 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 8368
I20260619 20:31:06.543884 8798 raft_consensus.cc:2229] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:31:06.544013 8798 raft_consensus.cc:2258] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:06.544378 8798 ts_tablet_manager.cc:1905] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:06.544600 8797 ts_tablet_manager.cc:1905] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:06.545995 8798 ts_tablet_manager.cc:1918] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:06.546079 8798 log.cc:1196] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/5cd2e866a2404c78b16379ba988ae374
I20260619 20:31:06.546317 8798 ts_tablet_manager.cc:1939] T 5cd2e866a2404c78b16379ba988ae374 P 9b3a9e96c5834ba2900e2201ec557eed: Deleting consensus metadata
I20260619 20:31:06.546531 8797 ts_tablet_manager.cc:1918] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:06.546604 8797 log.cc:1196] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/5cd2e866a2404c78b16379ba988ae374
I20260619 20:31:06.546729 8294 catalog_manager.cc:4841] TS 9b3a9e96c5834ba2900e2201ec557eed (127.21.151.3:36037): tablet 5cd2e866a2404c78b16379ba988ae374 (table test-table [id=23f3764840fa421f9dffd3e08e16129c]) successfully deleted
I20260619 20:31:06.546845 8797 ts_tablet_manager.cc:1939] T 5cd2e866a2404c78b16379ba988ae374 P 614dc2c2391e4a81b561cf65e5fb5c19: Deleting consensus metadata
I20260619 20:31:06.547407 8292 catalog_manager.cc:4841] TS 614dc2c2391e4a81b561cf65e5fb5c19 (127.21.151.2:42121): tablet 5cd2e866a2404c78b16379ba988ae374 (table test-table [id=23f3764840fa421f9dffd3e08e16129c]) successfully deleted
W20260619 20:31:06.548027 8295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:46755 (error 108)
I20260619 20:31:06.548287 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 8499
I20260619 20:31:06.548511 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:06.548393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:46755 (local address 127.0.0.1:35514)
0619 20:31:06.548439 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:06.548474 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 20:31:06.548609 8295 catalog_manager.cc:4584] TS 6341ffc93ec94a95861dce16480fc6b2 (127.21.151.1:46755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cd2e866a2404c78b16379ba988ae374: Network error: Client connection negotiation failed: client connection to 127.21.151.1:46755: connect: Connection refused (error 111)
I20260619 20:31:06.554188 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 8630
I20260619 20:31:06.558846 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 8277
2026-06-19T20:31:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (852 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:06 dist-test-slave-f9rk krb5kdc[8802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:06 dist-test-slave-f9rk krb5kdc[8802](info): set up 2 sockets
Jun 19 20:31:06 dist-test-slave-f9rk krb5kdc[8802](info): commencing operation
krb5kdc: starting...
W20260619 20:31:07.604583 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:07 dist-test-slave-f9rk krb5kdc[8802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:07.714689 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44707
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:07.792598 8815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:07.792809 8815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:07.792855 8815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:07.793955 8815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:07.794015 8815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:07.794037 8815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:07.794059 8815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:07.794085 8815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:07.795521 8815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:44707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:44707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:07.795797 8815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:07.796025 8815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:07.800745 8820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:07.800834 8821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:07.800905 8815 server_base.cc:1034] running on GCE node
W20260619 20:31:07.800846 8823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:07.801273 8815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:07.801508 8815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:07.802672 8815 hybrid_clock.cc:648] HybridClock initialized: now 1781901067802665 us; error 36 us; skew 500 ppm
Jun 19 20:31:07 dist-test-slave-f9rk krb5kdc[8802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901067, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:07.805138 8815 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:07.805583 8815 webserver.cc:460] Webserver started at http://127.21.151.62:41253/ using document root <none> and password file <none>
I20260619 20:31:07.805832 8815 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:07.805897 8815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:07.806042 8815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:07.807029 8815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "f29bda96701a43ff9138508d32f5939e"
format_stamp: "Formatted at 2026-06-19 20:31:07 on dist-test-slave-f9rk"
server_key: "0368acf0c6cd79885e24073245d4e1c6"
server_key_iv: "f954e10fedb0d7b575cb6db5fc908a94"
server_key_version: "encryptionkey@0"
I20260619 20:31:07.807377 8815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "f29bda96701a43ff9138508d32f5939e"
format_stamp: "Formatted at 2026-06-19 20:31:07 on dist-test-slave-f9rk"
server_key: "0368acf0c6cd79885e24073245d4e1c6"
server_key_iv: "f954e10fedb0d7b575cb6db5fc908a94"
server_key_version: "encryptionkey@0"
I20260619 20:31:07.808756 8815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:31:07.809545 8830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:07.809752 8815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:07.809854 8815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "f29bda96701a43ff9138508d32f5939e"
format_stamp: "Formatted at 2026-06-19 20:31:07 on dist-test-slave-f9rk"
server_key: "0368acf0c6cd79885e24073245d4e1c6"
server_key_iv: "f954e10fedb0d7b575cb6db5fc908a94"
server_key_version: "encryptionkey@0"
I20260619 20:31:07.809938 8815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:07.830552 8815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:07.832945 8815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:07.833098 8815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:07.836726 8815 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:44707
I20260619 20:31:07.836774 8882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:44707 every 8 connection(s)
I20260619 20:31:07.837108 8815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:07.837927 8883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:07.838873 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 8815
I20260619 20:31:07.838994 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:07.839166 22108 external_mini_cluster.cc:1428] Setting key 294286daece753a2740e2d186ffecbec
I20260619 20:31:07.840454 8883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e: Bootstrap starting.
I20260619 20:31:07.841200 8883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:07.841496 8883 log.cc:824] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:07.842128 8883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e: No bootstrap required, opened a new log
I20260619 20:31:07.844440 8883 raft_consensus.cc:348] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29bda96701a43ff9138508d32f5939e" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44707 } }
I20260619 20:31:07.844610 8883 raft_consensus.cc:374] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:07.844650 8883 raft_consensus.cc:729] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29bda96701a43ff9138508d32f5939e, State: Initialized, Role: FOLLOWER
I20260619 20:31:07.844784 8883 consensus_queue.cc:260] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29bda96701a43ff9138508d32f5939e" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44707 } }
I20260619 20:31:07.844868 8883 raft_consensus.cc:388] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:07.844903 8883 raft_consensus.cc:482] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:07.844959 8883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:07.845615 8883 raft_consensus.cc:504] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29bda96701a43ff9138508d32f5939e" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44707 } }
I20260619 20:31:07.845696 8883 leader_election.cc:302] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f29bda96701a43ff9138508d32f5939e; no voters:
I20260619 20:31:07.845866 8883 leader_election.cc:288] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [CANDIDATE]: Term 1 election: Requested vote from peers
Jun 19 20:31:07 dist-test-slave-f9rk krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:07.845903 8888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:07.846045 8888 raft_consensus.cc:686] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [term 1 LEADER]: Becoming Leader. State: Replica: f29bda96701a43ff9138508d32f5939e, State: Running, Role: LEADER
I20260619 20:31:07.846107 8883 sys_catalog.cc:564] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:07.846135 8888 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29bda96701a43ff9138508d32f5939e" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44707 } }
I20260619 20:31:07.846447 8888 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f29bda96701a43ff9138508d32f5939e. Latest consensus state: current_term: 1 leader_uuid: "f29bda96701a43ff9138508d32f5939e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29bda96701a43ff9138508d32f5939e" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44707 } } }
I20260619 20:31:07.846529 8888 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:07.846458 8889 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f29bda96701a43ff9138508d32f5939e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29bda96701a43ff9138508d32f5939e" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 44707 } } }
I20260619 20:31:07.847028 8889 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:07.847285 8894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:07.847769 8894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:07.849160 8894 catalog_manager.cc:1347] Generated new cluster ID: ce650b015ed04e65a52306c5a6f16bbc
I20260619 20:31:07.849231 8894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:07.850474 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:07.839687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.21.151.62:44707)
0619 20:31:07.839875 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:07.839882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:07.839907 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:31:07.840325 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:07.840331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:07.840376 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:07.840505 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:07.841334 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:07.842263 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:07.843264 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:07.843709 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:07.843791 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:07.846499 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:07.846504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:07.846505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:07.846515 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:07.848815 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:07.849257 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:07.849260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:07.849261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:07.849371 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:07.849588 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:07.849589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:07.849590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:07.849893 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:07.850035 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:07.850149 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 20:31:07.850284 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260619 20:31:07.868047 8894 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:07.868782 8894 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:07.874505 8894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e: Generated new TSK 0
I20260619 20:31:07.874694 8894 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.151.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
Entry for principal kudu/127.21.151.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab.
I20260619 20:31:07.906649 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44707
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:07.985278 8909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:07.985507 8909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:07.985543 8909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:07.986620 8909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:07.986699 8909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:07.986742 8909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:31:07.988137 8909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.1.keytab
--principal=kudu/127.21.151.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:07.988384 8909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:07.988623 8909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:07.993510 8917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:07.993494 8914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:07.993511 8915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:07.993810 8909 server_base.cc:1034] running on GCE node
I20260619 20:31:07.993966 8909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:07.994144 8909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:07.995375 8909 hybrid_clock.cc:648] HybridClock initialized: now 1781901067995366 us; error 34 us; skew 500 ppm
Jun 19 20:31:07 dist-test-slave-f9rk krb5kdc[8802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901067, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:07.997961 8909 init.cc:377] Logged in from keytab as kudu/127.21.151.1@KRBTEST.COM (short username kudu)
I20260619 20:31:07.998399 8909 webserver.cc:460] Webserver started at http://127.21.151.1:46397/ using document root <none> and password file <none>
I20260619 20:31:07.998627 8909 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:07.998695 8909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:07.998848 8909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:07.999943 8909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "05238939aefe4c7da46a065e8e3fac5d"
format_stamp: "Formatted at 2026-06-19 20:31:07 on dist-test-slave-f9rk"
server_key: "2f7caf5de2adc937d886041bef569b66"
server_key_iv: "7c1efe9c9ad81b48f95970cab681a474"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.000330 8909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "05238939aefe4c7da46a065e8e3fac5d"
format_stamp: "Formatted at 2026-06-19 20:31:07 on dist-test-slave-f9rk"
server_key: "2f7caf5de2adc937d886041bef569b66"
server_key_iv: "7c1efe9c9ad81b48f95970cab681a474"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.001701 8909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:31:08.002379 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.002549 8909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:08.002640 8909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "05238939aefe4c7da46a065e8e3fac5d"
format_stamp: "Formatted at 2026-06-19 20:31:07 on dist-test-slave-f9rk"
server_key: "2f7caf5de2adc937d886041bef569b66"
server_key_iv: "7c1efe9c9ad81b48f95970cab681a474"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.002718 8909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:08.019667 8909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:08.021958 8909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:08.022080 8909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:08.022257 8909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:08.022622 8909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:08.022660 8909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.022682 8909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:08.022730 8909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.028633 8909 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:45077
I20260619 20:31:08.028684 9037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:45077 every 8 connection(s)
I20260619 20:31:08.029004 8909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:31:08.031263 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 8909
I20260619 20:31:08.031399 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:31:08.031581 22108 external_mini_cluster.cc:1428] Setting key 05568577c887e31df2ac2e31c57cb14c
Jun 19 20:31:08 dist-test-slave-f9rk krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901067, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.1@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:08.039083 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.029666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:32819 (local address 127.21.151.62:44707)
0619 20:31:08.029709 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.029711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.029927 (+ 216us) server_negotiation.cc:407] Connection header received
0619 20:31:08.030400 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.030401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.030410 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.030462 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:08.031397 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.032115 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.033172 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.033520 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.033600 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.036175 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:08.036176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:08.036177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:08.036183 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:08.037629 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:08.038110 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:08.038111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:08.038111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:08.038162 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:08.038454 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:08.038454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:08.038455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:08.038549 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:08.038597 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.038937 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.039016 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:31:08.039359 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.029648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44707 (local address 127.21.151.1:32819)
0619 20:31:08.029798 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:08.029834 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:08.030312 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:08.030515 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:08.030518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:08.031092 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:08.031286 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.031290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.032233 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.032234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.033076 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.033078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.033618 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.033619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.033777 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.034311 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:08.034322 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:08.036077 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:08.037717 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:08.037726 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:08.037728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:08.038024 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:08.038226 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:08.038226 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:08.038227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:08.038384 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:08.038616 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:08.038622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:08.038788 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 20:31:08.038915 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 20:31:08.039088 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.21.151.2@KRBTEST.COM; defaulting to no policy
I20260619 20:31:08.039940 9038 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44707
I20260619 20:31:08.040066 9038 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:08.040277 9038 heartbeater.cc:511] Master 127.21.151.62:44707 requested a full tablet report, sending...
I20260619 20:31:08.040838 8847 ts_manager.cc:194] Registered new tserver with Master: 05238939aefe4c7da46a065e8e3fac5d (127.21.151.1:45077)
I20260619 20:31:08.041471 8847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.1@KRBTEST.COM'} at 127.21.151.1:32819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
Entry for principal kudu/127.21.151.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab.
I20260619 20:31:08.056315 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44707
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:08.135702 9043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:08.135890 9043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:08.135912 9043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:08.136960 9043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:08.136991 9043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:08.137015 9043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:31:08.138352 9043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.2.keytab
--principal=kudu/127.21.151.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:08.138538 9043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:08.138697 9043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:08.143422 9049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:08.143431 9051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:08.143431 9048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:08.143573 9043 server_base.cc:1034] running on GCE node
I20260619 20:31:08.143883 9043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:08.144101 9043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:08.145253 9043 hybrid_clock.cc:648] HybridClock initialized: now 1781901068145246 us; error 26 us; skew 500 ppm
Jun 19 20:31:08 dist-test-slave-f9rk krb5kdc[8802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901068, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:08.147601 9043 init.cc:377] Logged in from keytab as kudu/127.21.151.2@KRBTEST.COM (short username kudu)
I20260619 20:31:08.148135 9043 webserver.cc:460] Webserver started at http://127.21.151.2:45835/ using document root <none> and password file <none>
I20260619 20:31:08.148386 9043 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:08.148452 9043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:08.148594 9043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:08.149608 9043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "40457c3b0fcb45c981b94ef9c8b0c322"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "5778c9c33527b9c89b723204cb19cbef"
server_key_iv: "df0ac2230a2178af583b6aebb414631d"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.149956 9043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "40457c3b0fcb45c981b94ef9c8b0c322"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "5778c9c33527b9c89b723204cb19cbef"
server_key_iv: "df0ac2230a2178af583b6aebb414631d"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.151219 9043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 20:31:08.152014 9058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.152179 9043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.152256 9043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "40457c3b0fcb45c981b94ef9c8b0c322"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "5778c9c33527b9c89b723204cb19cbef"
server_key_iv: "df0ac2230a2178af583b6aebb414631d"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.152307 9043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:08.161026 9043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:08.162681 9043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:08.162781 9043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:08.162945 9043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:08.163170 9043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:08.163198 9043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.163218 9043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:08.163229 9043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.168948 9043 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:34051
I20260619 20:31:08.168987 9171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:34051 every 8 connection(s)
I20260619 20:31:08.169305 9043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:31:08.170769 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 9043
I20260619 20:31:08.170900 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:31:08.171085 22108 external_mini_cluster.cc:1428] Setting key 7d52e3e91f0d93e2b158182ee133e1c5
Jun 19 20:31:08 dist-test-slave-f9rk krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901068, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.2@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:08.179446 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.169983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:50833 (local address 127.21.151.62:44707)
0619 20:31:08.170039 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.170041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.170246 (+ 205us) server_negotiation.cc:407] Connection header received
0619 20:31:08.170787 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.170788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.170801 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.170868 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:08.171965 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.172444 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.173821 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.174014 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.174072 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.176481 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:08.176483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:08.176483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:08.176491 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:08.178079 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:08.178469 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:08.178470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:08.178471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:08.178533 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:08.178751 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:08.178752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:08.178752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:08.178891 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:08.178970 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.179236 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.179322 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:31:08.179459 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.169959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44707 (local address 127.21.151.2:50833)
0619 20:31:08.170130 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:08.170168 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:08.170684 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:08.170938 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:08.170942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:08.171586 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:08.171874 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.171879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.172539 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.172540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.173725 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.173728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.174093 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.174094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.174237 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.174718 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:08.174728 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:08.176382 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:08.178170 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:08.178175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:08.178177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:08.178396 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:08.178603 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:08.178604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:08.178604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:08.178688 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:08.178983 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:08.178988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:08.179110 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 20:31:08.179189 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:31:08.179303 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.21.151.3@KRBTEST.COM; defaulting to no policy
I20260619 20:31:08.179924 9172 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44707
I20260619 20:31:08.180016 9172 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:08.180217 9172 heartbeater.cc:511] Master 127.21.151.62:44707 requested a full tablet report, sending...
I20260619 20:31:08.180583 8847 ts_manager.cc:194] Registered new tserver with Master: 40457c3b0fcb45c981b94ef9c8b0c322 (127.21.151.2:34051)
I20260619 20:31:08.180909 8847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.2@KRBTEST.COM'} at 127.21.151.2:50833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
Entry for principal kudu/127.21.151.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab.
I20260619 20:31:08.196552 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44707
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:43411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:08.278311 9177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:08.278501 9177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:08.278522 9177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:08.279685 9177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:08.279723 9177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:08.279749 9177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:31:08.281208 9177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:43411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.3.keytab
--principal=kudu/127.21.151.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:08.281390 9177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:08.281561 9177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:08.286512 9185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:08.286672 9177 server_base.cc:1034] running on GCE node
W20260619 20:31:08.286517 9183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:08.286523 9182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:08.286939 9177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:08.287171 9177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:08.288327 9177 hybrid_clock.cc:648] HybridClock initialized: now 1781901068288309 us; error 30 us; skew 500 ppm
Jun 19 20:31:08 dist-test-slave-f9rk krb5kdc[8802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901068, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:08.290720 9177 init.cc:377] Logged in from keytab as kudu/127.21.151.3@KRBTEST.COM (short username kudu)
I20260619 20:31:08.291079 9177 webserver.cc:460] Webserver started at http://127.21.151.3:41359/ using document root <none> and password file <none>
I20260619 20:31:08.291262 9177 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:08.291314 9177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:08.291436 9177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:08.292493 9177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "c89ad81fc35f4dc98d9cda7ef8432321"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "85dc4b3e1810458e532f5a4a5bd398b8"
server_key_iv: "d18506e33efa0761a83cb6675456ff9f"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.292817 9177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "c89ad81fc35f4dc98d9cda7ef8432321"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "85dc4b3e1810458e532f5a4a5bd398b8"
server_key_iv: "d18506e33efa0761a83cb6675456ff9f"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.294029 9177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:08.294690 9192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.294863 9177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:08.294960 9177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "c89ad81fc35f4dc98d9cda7ef8432321"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "85dc4b3e1810458e532f5a4a5bd398b8"
server_key_iv: "d18506e33efa0761a83cb6675456ff9f"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.295013 9177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:08.307335 9177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:08.309355 9177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:08.309520 9177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:08.309734 9177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:08.309983 9177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:08.310014 9177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.310034 9177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:08.310047 9177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.315850 9177 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:39589
I20260619 20:31:08.315954 9305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:39589 every 8 connection(s)
I20260619 20:31:08.316244 9177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:31:08.321233 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 9177
I20260619 20:31:08.321360 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:31:08.321563 22108 external_mini_cluster.cc:1428] Setting key aff66114323a6fa47905706071f9b292
Jun 19 20:31:08 dist-test-slave-f9rk krb5kdc[8802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901068, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.3@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:08.325317 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.316967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:43287 (local address 127.21.151.62:44707)
0619 20:31:08.317010 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.317013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.317196 (+ 183us) server_negotiation.cc:407] Connection header received
0619 20:31:08.317705 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.317706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.317718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.317771 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:08.318674 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.319139 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.320226 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.320436 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.320540 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.322815 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:08.322816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:08.322817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:08.322824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:08.323981 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:08.324397 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:08.324398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:08.324399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:08.324456 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:08.324708 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:08.324709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:08.324709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:08.324816 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:08.324892 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.325152 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.325227 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:31:08.325518 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.316934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:44707 (local address 127.21.151.3:43287)
0619 20:31:08.317070 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:08.317114 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:08.317603 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:08.317836 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:08.317840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:08.318392 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:08.318580 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.318584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.319226 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.319227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.320100 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.320103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.320548 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.320549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.320672 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.321187 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:08.321195 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:08.322726 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:08.324071 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:08.324077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:08.324080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:08.324307 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:08.324523 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:08.324524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:08.324525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:08.324631 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:08.324884 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:08.324890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:08.325025 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 20:31:08.325131 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 20:31:08.325284 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260619 20:31:08.325937 9306 heartbeater.cc:347] Connected to a master server at 127.21.151.62:44707
I20260619 20:31:08.326025 9306 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:08.326152 9306 heartbeater.cc:511] Master 127.21.151.62:44707 requested a full tablet report, sending...
I20260619 20:31:08.326489 8847 ts_manager.cc:194] Registered new tserver with Master: c89ad81fc35f4dc98d9cda7ef8432321 (127.21.151.3:39589)
I20260619 20:31:08.326843 8847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.151.3@KRBTEST.COM'} at 127.21.151.3:43287
I20260619 20:31:08.333271 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:31:08.339784 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.334523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.21.151.62:44707)
0619 20:31:08.334555 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.334557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.334717 (+ 160us) server_negotiation.cc:407] Connection header received
0619 20:31:08.334737 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.334738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.334747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.334784 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:08.335280 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.335816 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.336485 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.336664 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.336710 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.337700 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:08.337701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:08.337701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:08.337707 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:08.338816 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:08.339099 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:08.339100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:08.339100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:08.339145 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:08.339282 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:08.339283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:08.339283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:08.339400 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:08.339475 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.339564 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.339671 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:31:08.341188 8847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:31:08.342656 8847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:31:08.349243 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.344929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.21.151.3:39589)
0619 20:31:08.345212 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.345216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.345226 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:31:08.345263 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.345265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.345282 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.345359 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:08.345766 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.346599 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.348186 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.348795 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.348879 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.348967 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.349120 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.349160 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":77,"threads_started":1}
I20260619 20:31:08.349232 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.344773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:39589 (local address 127.0.0.1:47146)
0619 20:31:08.344876 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:08.344912 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:08.344974 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:08.345361 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:08.345364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:08.345365 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:08.345427 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.345429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.346703 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.346704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.348091 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.348094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.348879 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.348880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.348988 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.349002 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 20:31:08.349102 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 20:31:08.349159 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260619 20:31:08.349403 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.345024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.21.151.1:45077)
0619 20:31:08.345507 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.345516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.345526 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:31:08.345583 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.345585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.345599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.345676 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:08.345818 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.346980 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.348177 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.348928 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.348996 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.349048 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.349289 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.349326 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":70,"threads_started":1}
I20260619 20:31:08.349409 9318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.345010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.21.151.2:34051)
0619 20:31:08.345168 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.345172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.345239 (+ 67us) server_negotiation.cc:407] Connection header received
0619 20:31:08.345314 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.345316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.345326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.345411 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:08.345539 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.346769 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.348211 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.348925 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.348996 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.349049 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.349289 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.349327 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260619 20:31:08.349601 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.344778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:45077 (local address 127.0.0.1:59420)
0619 20:31:08.345072 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:08.345081 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:08.345121 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:08.345687 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:08.345687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:08.345689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:08.345730 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.345731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.347150 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.347151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.348088 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.348090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.349085 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.349086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.349151 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.349155 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:31:08.349205 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:31:08.349247 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":3,"thread_start_us":56,"threads_started":1}
I20260619 20:31:08.349602 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.344785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:34051 (local address 127.0.0.1:58588)
0619 20:31:08.345171 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:08.345183 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:08.345248 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:08.345423 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:08.345425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:08.345427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:08.345479 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.345480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.346868 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.346869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.348150 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.348152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.349086 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.349086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.349151 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.349155 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 20:31:08.349205 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 20:31:08.349247 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":5,"thread_start_us":51,"threads_started":1}
I20260619 20:31:08.350457 9240 tablet_service.cc:1467] Processing CreateTablet for tablet a2887161603b426bae9fc35e6dee29ba (DEFAULT_TABLE table=test-table [id=4e4219bc1606477c9a204f3ece925652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:08.350457 9106 tablet_service.cc:1467] Processing CreateTablet for tablet a2887161603b426bae9fc35e6dee29ba (DEFAULT_TABLE table=test-table [id=4e4219bc1606477c9a204f3ece925652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:08.350788 9240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2887161603b426bae9fc35e6dee29ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:08.350803 9106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2887161603b426bae9fc35e6dee29ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:08.353487 9320 tablet_bootstrap.cc:492] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: Bootstrap starting.
I20260619 20:31:08.353488 9321 tablet_bootstrap.cc:492] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: Bootstrap starting.
I20260619 20:31:08.353951 8972 tablet_service.cc:1467] Processing CreateTablet for tablet a2887161603b426bae9fc35e6dee29ba (DEFAULT_TABLE table=test-table [id=4e4219bc1606477c9a204f3ece925652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:08.354136 9320 tablet_bootstrap.cc:654] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:08.354194 9321 tablet_bootstrap.cc:654] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:08.354287 8972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2887161603b426bae9fc35e6dee29ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:08.354703 9321 log.cc:824] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:08.354727 9320 log.cc:824] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:08.355384 9321 tablet_bootstrap.cc:492] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: No bootstrap required, opened a new log
I20260619 20:31:08.355469 9321 ts_tablet_manager.cc:1397] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:31:08.356333 9320 tablet_bootstrap.cc:492] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: No bootstrap required, opened a new log
I20260619 20:31:08.356456 9320 ts_tablet_manager.cc:1397] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 20:31:08.357262 9324 tablet_bootstrap.cc:492] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d: Bootstrap starting.
I20260619 20:31:08.357244 9321 raft_consensus.cc:348] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 } } peers { permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 } } peers { permanent_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34051 } }
I20260619 20:31:08.357422 9321 raft_consensus.cc:374] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:08.357466 9321 raft_consensus.cc:729] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89ad81fc35f4dc98d9cda7ef8432321, State: Initialized, Role: FOLLOWER
I20260619 20:31:08.357596 9321 consensus_queue.cc:260] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 } } peers { permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 } } peers { permanent_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34051 } }
I20260619 20:31:08.357872 9321 ts_tablet_manager.cc:1428] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:31:08.357903 9324 tablet_bootstrap.cc:654] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:08.358033 9306 heartbeater.cc:503] Master 127.21.151.62:44707 was elected leader, sending a full tablet report...
I20260619 20:31:08.358587 9324 log.cc:824] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:08.358629 9320 raft_consensus.cc:348] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 } } peers { permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 } } peers { permanent_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34051 } }
I20260619 20:31:08.358749 9320 raft_consensus.cc:374] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:08.358768 9320 raft_consensus.cc:729] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40457c3b0fcb45c981b94ef9c8b0c322, State: Initialized, Role: FOLLOWER
I20260619 20:31:08.358916 9320 consensus_queue.cc:260] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 } } peers { permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 } } peers { permanent_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34051 } }
I20260619 20:31:08.359184 9320 ts_tablet_manager.cc:1428] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 20:31:08.359323 9172 heartbeater.cc:503] Master 127.21.151.62:44707 was elected leader, sending a full tablet report...
I20260619 20:31:08.359386 9324 tablet_bootstrap.cc:492] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d: No bootstrap required, opened a new log
I20260619 20:31:08.359465 9324 ts_tablet_manager.cc:1397] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 20:31:08.361259 9324 raft_consensus.cc:348] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 } } peers { permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 } } peers { permanent_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34051 } }
I20260619 20:31:08.361408 9324 raft_consensus.cc:374] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:08.361465 9324 raft_consensus.cc:729] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05238939aefe4c7da46a065e8e3fac5d, State: Initialized, Role: FOLLOWER
I20260619 20:31:08.361572 9324 consensus_queue.cc:260] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 } } peers { permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 } } peers { permanent_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34051 } }
I20260619 20:31:08.361840 9324 ts_tablet_manager.cc:1428] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 20:31:08.361873 9038 heartbeater.cc:503] Master 127.21.151.62:44707 was elected leader, sending a full tablet report...
W20260619 20:31:08.419806 9173 tablet.cc:2369] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:31:08.529829 9039 tablet.cc:2369] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 20:31:08.566877 9307 tablet.cc:2369] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 20:31:08.645861 9327 raft_consensus.cc:482] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:31:08.645989 9327 raft_consensus.cc:504] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 } } peers { permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 } } peers { permanent_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34051 } }
I20260619 20:31:08.646445 9327 leader_election.cc:288] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05238939aefe4c7da46a065e8e3fac5d (127.21.151.1:45077), c89ad81fc35f4dc98d9cda7ef8432321 (127.21.151.3:39589)
I20260619 20:31:08.650573 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.646742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:42719 (local address 127.21.151.3:39589)
0619 20:31:08.646798 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.646801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.646811 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:31:08.646841 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.646842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.646853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.646973 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:08.647198 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.648021 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.649607 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.650188 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.650271 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.650279 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.650464 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.650501 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 20:31:08.650573 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.646678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:41841 (local address 127.21.151.1:45077)
0619 20:31:08.646737 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.646739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.646972 (+ 233us) server_negotiation.cc:407] Connection header received
0619 20:31:08.647073 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.647074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.647084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.647122 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:08.647299 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.648044 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.649607 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.650209 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.650271 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.650279 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.650464 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.650501 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 20:31:08.650583 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.646623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:39589 (local address 127.21.151.2:42719)
0619 20:31:08.646678 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:08.646689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:08.646732 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:08.646998 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:08.647000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:08.647001 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:08.647095 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.647097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.648138 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.648139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.649512 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.649515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.650287 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.650288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.650372 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.650377 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:31:08.650436 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:31:08.650486 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":134}
I20260619 20:31:08.650589 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.646635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:45077 (local address 127.21.151.2:41841)
0619 20:31:08.646893 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:08.646921 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:08.646976 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:08.647147 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:08.647148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:08.647149 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:08.647208 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.647209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.648138 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.648139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.649512 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:08.649515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.650287 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.650288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:08.650372 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.650377 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 20:31:08.650436 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:31:08.650486 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":56,"thread_start_us":140,"threads_started":1}
I20260619 20:31:08.650974 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2887161603b426bae9fc35e6dee29ba" candidate_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05238939aefe4c7da46a065e8e3fac5d" is_pre_election: true
I20260619 20:31:08.650956 9260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2887161603b426bae9fc35e6dee29ba" candidate_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" is_pre_election: true
I20260619 20:31:08.651115 9260 raft_consensus.cc:2454] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40457c3b0fcb45c981b94ef9c8b0c322 in term 0.
I20260619 20:31:08.651115 8992 raft_consensus.cc:2454] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40457c3b0fcb45c981b94ef9c8b0c322 in term 0.
I20260619 20:31:08.651340 9062 leader_election.cc:302] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40457c3b0fcb45c981b94ef9c8b0c322, c89ad81fc35f4dc98d9cda7ef8432321; no voters:
I20260619 20:31:08.651516 9327 raft_consensus.cc:2781] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:31:08.651593 9327 raft_consensus.cc:482] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:31:08.651647 9327 raft_consensus.cc:3037] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:08.652396 9327 raft_consensus.cc:504] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 } } peers { permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 } } peers { permanent_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34051 } }
I20260619 20:31:08.652539 9327 leader_election.cc:288] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [CANDIDATE]: Term 1 election: Requested vote from peers 05238939aefe4c7da46a065e8e3fac5d (127.21.151.1:45077), c89ad81fc35f4dc98d9cda7ef8432321 (127.21.151.3:39589)
I20260619 20:31:08.652689 8992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2887161603b426bae9fc35e6dee29ba" candidate_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05238939aefe4c7da46a065e8e3fac5d"
I20260619 20:31:08.652772 8992 raft_consensus.cc:3037] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:08.653478 8992 raft_consensus.cc:2454] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40457c3b0fcb45c981b94ef9c8b0c322 in term 1.
I20260619 20:31:08.653600 9260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2887161603b426bae9fc35e6dee29ba" candidate_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89ad81fc35f4dc98d9cda7ef8432321"
I20260619 20:31:08.653661 9059 leader_election.cc:302] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05238939aefe4c7da46a065e8e3fac5d, 40457c3b0fcb45c981b94ef9c8b0c322; no voters:
I20260619 20:31:08.653683 9260 raft_consensus.cc:3037] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:08.653748 9327 raft_consensus.cc:2781] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:08.653893 9327 raft_consensus.cc:686] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 1 LEADER]: Becoming Leader. State: Replica: 40457c3b0fcb45c981b94ef9c8b0c322, State: Running, Role: LEADER
I20260619 20:31:08.654004 9327 consensus_queue.cc:237] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 } } peers { permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 } } peers { permanent_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34051 } }
I20260619 20:31:08.654621 9260 raft_consensus.cc:2454] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40457c3b0fcb45c981b94ef9c8b0c322 in term 1.
I20260619 20:31:08.654955 8846 catalog_manager.cc:5495] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40457c3b0fcb45c981b94ef9c8b0c322 (127.21.151.2). New cstate: current_term: 1 leader_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40457c3b0fcb45c981b94ef9c8b0c322" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 34051 } health_report { overall_health: HEALTHY } } }
I20260619 20:31:08.701088 9318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.698940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.21.151.2:34051)
0619 20:31:08.699005 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.699007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.699014 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:31:08.699048 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.699049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.699067 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.699128 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:08.699229 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.699747 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.700457 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.700634 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.700683 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.700811 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:08.700904 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:08.700969 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.700996 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.701012 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 20:31:08.702831 8992 raft_consensus.cc:1261] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d [term 1 FOLLOWER]: Refusing update from remote peer 40457c3b0fcb45c981b94ef9c8b0c322: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:08.702829 9260 raft_consensus.cc:1261] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321 [term 1 FOLLOWER]: Refusing update from remote peer 40457c3b0fcb45c981b94ef9c8b0c322: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:08.703181 9330 consensus_queue.cc:1035] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c89ad81fc35f4dc98d9cda7ef8432321" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 39589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:08.703315 9327 consensus_queue.cc:1035] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05238939aefe4c7da46a065e8e3fac5d" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 45077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:08.705996 9335 mvcc.cc:205] Tried to move back new op lower bound from 7298666777404116992 to 7298666777207435264. Current Snapshot: MvccSnapshot[applied={T|T < 7298666777404116992}]
I20260619 20:31:08.706142 9334 mvcc.cc:205] Tried to move back new op lower bound from 7298666777404116992 to 7298666777207435264. Current Snapshot: MvccSnapshot[applied={T|T < 7298666777404116992}]
I20260619 20:31:08.708218 8846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:08.708312 8846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:08.709059 8846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f29bda96701a43ff9138508d32f5939e: Sending DeleteTablet for 3 replicas of tablet a2887161603b426bae9fc35e6dee29ba
I20260619 20:31:08.709373 8972 tablet_service.cc:1514] Processing DeleteTablet for tablet a2887161603b426bae9fc35e6dee29ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:08 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:59420
I20260619 20:31:08.709442 9240 tablet_service.cc:1514] Processing DeleteTablet for tablet a2887161603b426bae9fc35e6dee29ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:08 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:47146
I20260619 20:31:08.709583 9106 tablet_service.cc:1514] Processing DeleteTablet for tablet a2887161603b426bae9fc35e6dee29ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:08 UTC) from {username='kudu', principal='kudu/127.21.151.62@KRBTEST.COM'} at 127.0.0.1:58588
I20260619 20:31:08.709671 9342 tablet_replica.cc:330] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d: stopping tablet replica
I20260619 20:31:08.709764 9342 raft_consensus.cc:2229] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:31:08.709875 9342 raft_consensus.cc:2258] T a2887161603b426bae9fc35e6dee29ba P 05238939aefe4c7da46a065e8e3fac5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:08.709946 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 8909
I20260619 20:31:08.711133 9343 tablet_replica.cc:330] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: stopping tablet replica
I20260619 20:31:08.711247 9343 raft_consensus.cc:2229] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:31:08.711361 9343 raft_consensus.cc:2258] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:08.711558 9344 tablet_replica.cc:330] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: stopping tablet replica
I20260619 20:31:08.711797 9344 raft_consensus.cc:2229] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:31:08.711892 9344 raft_consensus.cc:2258] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:08.711997 9343 ts_tablet_manager.cc:1905] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:08.712308 9344 ts_tablet_manager.cc:1905] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:08.713948 9343 ts_tablet_manager.cc:1918] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:08.714089 9343 log.cc:1196] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/a2887161603b426bae9fc35e6dee29ba
I20260619 20:31:08.714103 9344 ts_tablet_manager.cc:1918] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:08.714156 9344 log.cc:1196] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/a2887161603b426bae9fc35e6dee29ba
I20260619 20:31:08.714457 9344 ts_tablet_manager.cc:1939] T a2887161603b426bae9fc35e6dee29ba P 40457c3b0fcb45c981b94ef9c8b0c322: Deleting consensus metadata
I20260619 20:31:08.714479 9343 ts_tablet_manager.cc:1939] T a2887161603b426bae9fc35e6dee29ba P c89ad81fc35f4dc98d9cda7ef8432321: Deleting consensus metadata
I20260619 20:31:08.714833 8832 catalog_manager.cc:4841] TS 40457c3b0fcb45c981b94ef9c8b0c322 (127.21.151.2:34051): tablet a2887161603b426bae9fc35e6dee29ba (table test-table [id=4e4219bc1606477c9a204f3ece925652]) successfully deleted
I20260619 20:31:08.714885 8834 catalog_manager.cc:4841] TS c89ad81fc35f4dc98d9cda7ef8432321 (127.21.151.3:39589): tablet a2887161603b426bae9fc35e6dee29ba (table test-table [id=4e4219bc1606477c9a204f3ece925652]) successfully deleted
W20260619 20:31:08.716073 8831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:45077 (error 108)
I20260619 20:31:08.716538 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.716393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:45077 (local address 127.0.0.1:59428)
0619 20:31:08.716449 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:08.716484 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 20:31:08.716600 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 9043
W20260619 20:31:08.716673 8831 catalog_manager.cc:4584] TS 05238939aefe4c7da46a065e8e3fac5d (127.21.151.1:45077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2887161603b426bae9fc35e6dee29ba: Network error: Client connection negotiation failed: client connection to 127.21.151.1:45077: connect: Connection refused (error 111)
I20260619 20:31:08.720515 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 9177
I20260619 20:31:08.724660 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 8815
2026-06-19T20:31:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2166 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T20:31:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:08Z Disabled control of system clock
I20260619 20:31:08.751575 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:39711
--webserver_interface=127.21.151.62
--webserver_port=0
--builtin_ntp_servers=127.21.151.20:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:31:08.831213 9349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:08.831418 9349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:08.831439 9349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:08.832544 9349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:08.832579 9349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:08.832592 9349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:08.832604 9349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:08.832616 9349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:08.833861 9349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:39711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:08.834050 9349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:08.834268 9349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:08.839113 9355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:08.839098 9357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:08.839165 9349 server_base.cc:1034] running on GCE node
W20260619 20:31:08.839232 9354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:08.839478 9349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:08.839807 9349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:08.840976 9349 hybrid_clock.cc:648] HybridClock initialized: now 1781901068840964 us; error 44 us; skew 500 ppm
I20260619 20:31:08.842167 9349 webserver.cc:460] Webserver started at http://127.21.151.62:44813/ using document root <none> and password file <none>
I20260619 20:31:08.842413 9349 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:08.842484 9349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:08.842631 9349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:08.843752 9349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "d336dee8eb2945778862dc136023a818"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "7d444c1036d5d45c7aab01594e54746c"
server_key_iv: "4235c92fac2cfec6d982ad5db758e8f5"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.844126 9349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "d336dee8eb2945778862dc136023a818"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "7d444c1036d5d45c7aab01594e54746c"
server_key_iv: "4235c92fac2cfec6d982ad5db758e8f5"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.845356 9349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 20:31:08.846040 9363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:08.846210 9349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:08.846299 9349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "d336dee8eb2945778862dc136023a818"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "7d444c1036d5d45c7aab01594e54746c"
server_key_iv: "4235c92fac2cfec6d982ad5db758e8f5"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.846380 9349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:08.863515 9349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:08.863931 9349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:08.864097 9349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:08.868309 9349 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:39711
I20260619 20:31:08.868338 9415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:39711 every 8 connection(s)
I20260619 20:31:08.868700 9349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:08.869683 9416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:08.871970 9416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818: Bootstrap starting.
I20260619 20:31:08.872475 9416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:08.872702 9416 log.cc:824] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:08.873308 9416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818: No bootstrap required, opened a new log
I20260619 20:31:08.874764 9416 raft_consensus.cc:348] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d336dee8eb2945778862dc136023a818" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39711 } }
I20260619 20:31:08.874895 9416 raft_consensus.cc:374] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:08.874917 9416 raft_consensus.cc:729] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d336dee8eb2945778862dc136023a818, State: Initialized, Role: FOLLOWER
I20260619 20:31:08.875048 9416 consensus_queue.cc:260] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d336dee8eb2945778862dc136023a818" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39711 } }
I20260619 20:31:08.875133 9416 raft_consensus.cc:388] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:08.875155 9416 raft_consensus.cc:482] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:08.875182 9416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:08.875677 9416 raft_consensus.cc:504] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d336dee8eb2945778862dc136023a818" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39711 } }
I20260619 20:31:08.875761 9416 leader_election.cc:302] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d336dee8eb2945778862dc136023a818; no voters:
I20260619 20:31:08.875859 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 9349
I20260619 20:31:08.875967 9416 leader_election.cc:288] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:08.876008 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:08.876045 9419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:08.876209 9419 raft_consensus.cc:686] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [term 1 LEADER]: Becoming Leader. State: Replica: d336dee8eb2945778862dc136023a818, State: Running, Role: LEADER
I20260619 20:31:08.876225 22108 external_mini_cluster.cc:1428] Setting key 576e663a1cfffe7650812b73647e5e46
I20260619 20:31:08.876231 9416 sys_catalog.cc:564] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:08.876310 9419 consensus_queue.cc:237] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d336dee8eb2945778862dc136023a818" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39711 } }
I20260619 20:31:08.876893 9420 sys_catalog.cc:455] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d336dee8eb2945778862dc136023a818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d336dee8eb2945778862dc136023a818" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39711 } } }
I20260619 20:31:08.876986 9420 sys_catalog.cc:458] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:08.877206 9421 sys_catalog.cc:455] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d336dee8eb2945778862dc136023a818. Latest consensus state: current_term: 1 leader_uuid: "d336dee8eb2945778862dc136023a818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d336dee8eb2945778862dc136023a818" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 39711 } } }
I20260619 20:31:08.877310 9421 sys_catalog.cc:458] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:08.877194 9430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:08.877841 9430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:08.879347 9430 catalog_manager.cc:1347] Generated new cluster ID: 7708127f89bc4c0da5fae30a4508f4f5
I20260619 20:31:08.879421 9430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:08.882215 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:08.876865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.21.151.62:39711)
0619 20:31:08.877074 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:08.877081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:08.877133 (+ 52us) server_negotiation.cc:407] Connection header received
0619 20:31:08.877247 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:08.877255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:08.877324 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:08.877528 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:08.877806 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.878957 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.880092 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:08.880727 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:08.880835 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:08.881531 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:08.881535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:08.881535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:08.881728 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:08.881765 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:08.881778 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:08.881838 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:08.881935 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 20:31:08.882012 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260619 20:31:08.890120 9430 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:08.890785 9430 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:08.903127 9430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818: Generated new TSK 0
I20260619 20:31:08.903326 9430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:08.905841 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.1:0
--local_ip_for_outbound_sockets=127.21.151.1
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39711
--builtin_ntp_servers=127.21.151.20:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:31:08.985698 9440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:08.985936 9440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:08.985975 9440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:08.987140 9440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:08.987203 9440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:08.987241 9440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.1
I20260619 20:31:08.988612 9440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.151.1
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.1
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:08.988863 9440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:08.989185 9440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:08.994254 9448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:08.994268 9445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:08.994270 9446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:08.994601 9440 server_base.cc:1034] running on GCE node
I20260619 20:31:08.994781 9440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:08.995042 9440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:08.996203 9440 hybrid_clock.cc:648] HybridClock initialized: now 1781901068996184 us; error 34 us; skew 500 ppm
I20260619 20:31:08.997689 9440 webserver.cc:460] Webserver started at http://127.21.151.1:37215/ using document root <none> and password file <none>
I20260619 20:31:08.997949 9440 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:08.998049 9440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:08.998204 9440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:08.999255 9440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/data/instance:
uuid: "8de38b8fe705407b87ba8f58242b591e"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "1796f0138a3cef68d2ebbce7be66c517"
server_key_iv: "a7430116010adc4b0749e59ad8e6a3ed"
server_key_version: "encryptionkey@0"
I20260619 20:31:08.999681 9440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance:
uuid: "8de38b8fe705407b87ba8f58242b591e"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "1796f0138a3cef68d2ebbce7be66c517"
server_key_iv: "a7430116010adc4b0749e59ad8e6a3ed"
server_key_version: "encryptionkey@0"
I20260619 20:31:09.001258 9440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:31:09.002032 9454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:09.002200 9440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:09.002296 9440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/wal
uuid: "8de38b8fe705407b87ba8f58242b591e"
format_stamp: "Formatted at 2026-06-19 20:31:08 on dist-test-slave-f9rk"
server_key: "1796f0138a3cef68d2ebbce7be66c517"
server_key_iv: "a7430116010adc4b0749e59ad8e6a3ed"
server_key_version: "encryptionkey@0"
I20260619 20:31:09.002386 9440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:09.012789 9440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:09.013087 9440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:09.013230 9440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:09.013443 9440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:09.013749 9440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:09.013813 9440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:09.013840 9440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:09.013888 9440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:09.019271 9440 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.1:42123
I20260619 20:31:09.019352 9567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.1:42123 every 8 connection(s)
I20260619 20:31:09.019688 9440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/data/info.pb
I20260619 20:31:09.020320 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 9440
I20260619 20:31:09.020471 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-0/wal/instance
I20260619 20:31:09.020680 22108 external_mini_cluster.cc:1428] Setting key 3dbcda39a016c542f8c196cd944cef3d
I20260619 20:31:09.021757 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.2:0
--local_ip_for_outbound_sockets=127.21.151.2
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39711
--builtin_ntp_servers=127.21.151.20:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 20:31:09.025352 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.020471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:42869 (local address 127.21.151.62:39711)
0619 20:31:09.020529 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:09.020532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:09.020776 (+ 244us) server_negotiation.cc:407] Connection header received
0619 20:31:09.021408 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:09.021409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:09.021420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:09.021466 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:09.022369 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.022859 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.023930 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.024133 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.024216 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.025047 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:09.025048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:09.025049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:09.025054 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:09.025068 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:09.025073 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:09.025137 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:09.025232 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 20:31:09.025268 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:31:09.025549 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.020441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39711 (local address 127.21.151.1:42869)
0619 20:31:09.020637 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:09.020682 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:09.021319 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:09.021550 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:09.021555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:09.021565 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:09.022266 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.022275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.022957 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.022958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.023826 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.023829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.024223 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.024224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.024378 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.024906 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:31:09.024906 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:09.024933 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:31:09.024954 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:09.025139 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:09.025145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:09.025148 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:09.025235 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 20:31:09.025333 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260619 20:31:09.026036 9568 heartbeater.cc:347] Connected to a master server at 127.21.151.62:39711
I20260619 20:31:09.026141 9568 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:09.026347 9568 heartbeater.cc:511] Master 127.21.151.62:39711 requested a full tablet report, sending...
I20260619 20:31:09.026932 9371 ts_manager.cc:194] Registered new tserver with Master: 8de38b8fe705407b87ba8f58242b591e (127.21.151.1:42123)
I20260619 20:31:09.027936 9371 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.1:42869
W20260619 20:31:09.106392 9571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:09.106606 9571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:09.106627 9571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:09.107728 9571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:09.107760 9571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:09.107785 9571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.2
I20260619 20:31:09.109090 9571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.151.2
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.2
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:09.109275 9571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:09.109525 9571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:09.114475 9577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:09.114513 9576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:09.114428 9579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:09.114647 9571 server_base.cc:1034] running on GCE node
I20260619 20:31:09.114897 9571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:09.115157 9571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:09.116276 9571 hybrid_clock.cc:648] HybridClock initialized: now 1781901069116247 us; error 40 us; skew 500 ppm
I20260619 20:31:09.117816 9571 webserver.cc:460] Webserver started at http://127.21.151.2:32887/ using document root <none> and password file <none>
I20260619 20:31:09.118062 9571 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:09.118119 9571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:09.118274 9571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:09.119361 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/data/instance:
uuid: "959bc7bf651c45278e47960f2465715a"
format_stamp: "Formatted at 2026-06-19 20:31:09 on dist-test-slave-f9rk"
server_key: "f48456f713e614a92110f6180eee1c04"
server_key_iv: "dcb01b8dab873bc3c494558bee94a106"
server_key_version: "encryptionkey@0"
I20260619 20:31:09.119764 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance:
uuid: "959bc7bf651c45278e47960f2465715a"
format_stamp: "Formatted at 2026-06-19 20:31:09 on dist-test-slave-f9rk"
server_key: "f48456f713e614a92110f6180eee1c04"
server_key_iv: "dcb01b8dab873bc3c494558bee94a106"
server_key_version: "encryptionkey@0"
I20260619 20:31:09.121054 9571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:31:09.121798 9585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:09.121984 9571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:09.122079 9571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/wal
uuid: "959bc7bf651c45278e47960f2465715a"
format_stamp: "Formatted at 2026-06-19 20:31:09 on dist-test-slave-f9rk"
server_key: "f48456f713e614a92110f6180eee1c04"
server_key_iv: "dcb01b8dab873bc3c494558bee94a106"
server_key_version: "encryptionkey@0"
I20260619 20:31:09.122164 9571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:09.131419 9571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:09.131742 9571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:09.131889 9571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:09.132103 9571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:09.132426 9571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:09.132481 9571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:09.132520 9571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:09.132547 9571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:09.138080 9571 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.2:40575
I20260619 20:31:09.138118 9699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.2:40575 every 8 connection(s)
I20260619 20:31:09.138432 9571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/data/info.pb
I20260619 20:31:09.143196 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.139131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.2:38253 (local address 127.21.151.62:39711)
0619 20:31:09.139210 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:09.139212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:09.139390 (+ 178us) server_negotiation.cc:407] Connection header received
0619 20:31:09.139923 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:09.139924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:09.139939 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:09.139991 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:09.140608 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.141083 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.141984 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.142159 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.142206 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.142994 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:09.142995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:09.142995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:09.142998 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:09.143008 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:09.143011 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:09.143047 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:09.143120 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 20:31:09.143141 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 20:31:09.143378 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.139098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39711 (local address 127.21.151.2:38253)
0619 20:31:09.139277 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:09.139308 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:09.139840 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:09.140041 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:09.140044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:09.140052 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:09.140540 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.140544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.141155 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.141156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.141903 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.141905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.142227 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.142228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.142329 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.142869 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:31:09.142870 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:09.142896 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:31:09.142925 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:09.143049 (+ 124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:09.143052 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:09.143054 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:31:09.143098 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 20:31:09.143180 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260619 20:31:09.143890 9700 heartbeater.cc:347] Connected to a master server at 127.21.151.62:39711
I20260619 20:31:09.143982 9700 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:09.144115 9700 heartbeater.cc:511] Master 127.21.151.62:39711 requested a full tablet report, sending...
I20260619 20:31:09.144480 9371 ts_manager.cc:194] Registered new tserver with Master: 959bc7bf651c45278e47960f2465715a (127.21.151.2:40575)
I20260619 20:31:09.144802 9371 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.2:38253
I20260619 20:31:09.146600 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 9571
I20260619 20:31:09.146687 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/wal/instance
I20260619 20:31:09.146834 22108 external_mini_cluster.cc:1428] Setting key deae7cdd39cc3e830b3adc3224c4362e
I20260619 20:31:09.147607 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.151.3:0
--local_ip_for_outbound_sockets=127.21.151.3
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39711
--builtin_ntp_servers=127.21.151.20:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 20:31:09.228893 9703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:09.229118 9703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:09.229154 9703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:09.230291 9703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:09.230355 9703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:09.230399 9703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.151.3
I20260619 20:31:09.232174 9703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.151.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.151.3
--webserver_port=0
--tserver_master_addrs=127.21.151.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.151.3
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:09.232439 9703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:09.232740 9703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:09.239786 9708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:09.239753 9711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:09.240377 9703 server_base.cc:1034] running on GCE node
W20260619 20:31:09.240649 9709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:09.240988 9703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:09.241261 9703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:09.242449 9703 hybrid_clock.cc:648] HybridClock initialized: now 1781901069242428 us; error 36 us; skew 500 ppm
I20260619 20:31:09.244350 9703 webserver.cc:460] Webserver started at http://127.21.151.3:43631/ using document root <none> and password file <none>
I20260619 20:31:09.244680 9703 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:09.244762 9703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:09.244899 9703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:09.247095 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/data/instance:
uuid: "dfcccf65019644ed98dae498ede7503f"
format_stamp: "Formatted at 2026-06-19 20:31:09 on dist-test-slave-f9rk"
server_key: "68ffaaafcf5daff95f5d95186113049d"
server_key_iv: "8476dec2400145f143cb03091dcb12f5"
server_key_version: "encryptionkey@0"
I20260619 20:31:09.247607 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance:
uuid: "dfcccf65019644ed98dae498ede7503f"
format_stamp: "Formatted at 2026-06-19 20:31:09 on dist-test-slave-f9rk"
server_key: "68ffaaafcf5daff95f5d95186113049d"
server_key_iv: "8476dec2400145f143cb03091dcb12f5"
server_key_version: "encryptionkey@0"
I20260619 20:31:09.249678 9703 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260619 20:31:09.250624 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:09.250813 9703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:31:09.250901 9703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/wal
uuid: "dfcccf65019644ed98dae498ede7503f"
format_stamp: "Formatted at 2026-06-19 20:31:09 on dist-test-slave-f9rk"
server_key: "68ffaaafcf5daff95f5d95186113049d"
server_key_iv: "8476dec2400145f143cb03091dcb12f5"
server_key_version: "encryptionkey@0"
I20260619 20:31:09.250979 9703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:09.277711 9703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:09.278065 9703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:09.278227 9703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:09.278499 9703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 20:31:09.278879 9703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 20:31:09.278921 9703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:09.278954 9703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 20:31:09.278996 9703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:09.284741 9703 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.3:42639
I20260619 20:31:09.284845 9830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.3:42639 every 8 connection(s)
I20260619 20:31:09.285140 9703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/data/info.pb
I20260619 20:31:09.290246 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.285875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.3:57653 (local address 127.21.151.62:39711)
0619 20:31:09.285924 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:09.285926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:09.286136 (+ 210us) server_negotiation.cc:407] Connection header received
0619 20:31:09.286649 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:09.286650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:09.286660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:09.286712 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:09.287340 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.287900 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.289026 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.289202 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.289253 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.290021 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:09.290022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:09.290022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:09.290027 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:09.290040 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:09.290044 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:09.290086 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:09.290166 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 20:31:09.290188 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:31:09.290576 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.285863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:39711 (local address 127.21.151.3:57653)
0619 20:31:09.286038 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:09.286070 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:09.286547 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:09.286767 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:09.286770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:09.286777 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:09.287262 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.287266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.287977 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.287978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.288928 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.288932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.289292 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.289293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.289402 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.289879 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 20:31:09.289880 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:09.289907 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 20:31:09.289946 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:09.290098 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:09.290102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:09.290104 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 20:31:09.290157 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 20:31:09.290234 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260619 20:31:09.291070 9831 heartbeater.cc:347] Connected to a master server at 127.21.151.62:39711
I20260619 20:31:09.291167 9831 heartbeater.cc:464] Registering TS with master...
I20260619 20:31:09.291338 9831 heartbeater.cc:511] Master 127.21.151.62:39711 requested a full tablet report, sending...
I20260619 20:31:09.291724 9371 ts_manager.cc:194] Registered new tserver with Master: dfcccf65019644ed98dae498ede7503f (127.21.151.3:42639)
I20260619 20:31:09.292078 9371 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.151.3:57653
I20260619 20:31:09.292526 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 9703
I20260619 20:31:09.292644 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/wal/instance
I20260619 20:31:09.292817 22108 external_mini_cluster.cc:1428] Setting key 42d58085e57785d37577bf324b392eb7
I20260619 20:31:09.293730 22108 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 20:31:09.297514 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.295014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.21.151.62:39711)
0619 20:31:09.295078 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:09.295080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:09.295147 (+ 67us) server_negotiation.cc:407] Connection header received
0619 20:31:09.295195 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:09.295196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:09.295203 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:09.295245 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:09.295358 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.295868 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.296549 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.296715 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.296762 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.297306 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:09.297307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:09.297307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:09.297311 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:09.297321 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:09.297324 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:09.297373 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:09.297406 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 20:31:09.297422 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 20:31:09.299229 9371 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 20:31:09.300850 9371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 20:31:09.307070 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.302938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.21.151.2:40575)
0619 20:31:09.303295 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:09.303300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:09.303311 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:31:09.303352 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:09.303354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:09.303373 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:09.303468 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:09.303931 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.304703 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.306019 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.306640 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.306746 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.306846 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:09.306945 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 20:31:09.306982 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":101,"threads_started":1}
I20260619 20:31:09.307056 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.302830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:40575 (local address 127.0.0.1:50882)
0619 20:31:09.302979 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:09.303020 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:09.303118 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:09.303571 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:09.303573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:09.303575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:09.303669 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.303671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.304791 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.304792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.305927 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.305930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.306752 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.306753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.306836 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.306849 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 20:31:09.306931 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:31:09.306983 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":31,"thread_start_us":92,"threads_started":1}
I20260619 20:31:09.307427 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.302973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:42123 (local address 127.0.0.1:48354)
0619 20:31:09.303105 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:09.303115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:09.303158 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:09.303770 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:09.303770 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:09.303772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:09.303803 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.303804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.305090 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.305091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.306246 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.306248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.307191 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.307192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.307263 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.307266 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:09.307321 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 20:31:09.307358 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260619 20:31:09.307492 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.303310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.21.151.1:42123)
0619 20:31:09.303508 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:09.303511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:09.303521 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:31:09.303559 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:09.303561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:09.303574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:09.303766 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:09.303876 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.305009 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.306329 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.307024 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.307079 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.307152 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:09.307358 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 20:31:09.307401 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20260619 20:31:09.307960 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.303089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.21.151.3:42639)
0619 20:31:09.303323 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:09.303327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:09.303334 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:31:09.303361 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:09.303362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:09.303373 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:09.303488 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:09.303737 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.304819 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.306197 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.306934 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.307004 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.307088 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:09.307833 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 20:31:09.307875 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":66,"threads_started":1}
I20260619 20:31:09.307876 9634 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3b9bf7f3b040268cb1359fd18aaee4 (DEFAULT_TABLE table=test-table [id=4972824f5a99429fa4bdde4267bab108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:09.308271 9634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3b9bf7f3b040268cb1359fd18aaee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:09.308208 9502 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3b9bf7f3b040268cb1359fd18aaee4 (DEFAULT_TABLE table=test-table [id=4972824f5a99429fa4bdde4267bab108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:09.308451 9502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3b9bf7f3b040268cb1359fd18aaee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:09.310726 9845 tablet_bootstrap.cc:492] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e: Bootstrap starting.
I20260619 20:31:09.310849 9846 tablet_bootstrap.cc:492] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: Bootstrap starting.
I20260619 20:31:09.311262 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.302780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:42639 (local address 127.0.0.1:58258)
0619 20:31:09.303099 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:09.303110 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:09.303164 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:09.303560 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:09.303562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:09.303565 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:09.303654 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.303657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.304910 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.304911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.306115 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.306117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.307400 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.307400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.307739 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.307742 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 20:31:09.307818 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:31:09.311180 (+ 3362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260619 20:31:09.311407 9845 tablet_bootstrap.cc:654] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:09.311512 9846 tablet_bootstrap.cc:654] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:09.311769 9845 log.cc:824] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:09.311863 9846 log.cc:824] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:09.311975 9765 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3b9bf7f3b040268cb1359fd18aaee4 (DEFAULT_TABLE table=test-table [id=4972824f5a99429fa4bdde4267bab108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 20:31:09.312325 9765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3b9bf7f3b040268cb1359fd18aaee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:09.312552 9845 tablet_bootstrap.cc:492] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e: No bootstrap required, opened a new log
I20260619 20:31:09.312621 9845 ts_tablet_manager.cc:1397] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:31:09.315253 9845 raft_consensus.cc:348] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 } } peers { permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 } } peers { permanent_uuid: "8de38b8fe705407b87ba8f58242b591e" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42123 } }
I20260619 20:31:09.315431 9845 raft_consensus.cc:374] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:09.315469 9845 raft_consensus.cc:729] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de38b8fe705407b87ba8f58242b591e, State: Initialized, Role: FOLLOWER
I20260619 20:31:09.315585 9845 consensus_queue.cc:260] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 } } peers { permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 } } peers { permanent_uuid: "8de38b8fe705407b87ba8f58242b591e" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42123 } }
I20260619 20:31:09.315759 9849 tablet_bootstrap.cc:492] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: Bootstrap starting.
I20260619 20:31:09.315864 9845 ts_tablet_manager.cc:1428] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 20:31:09.316057 9568 heartbeater.cc:503] Master 127.21.151.62:39711 was elected leader, sending a full tablet report...
I20260619 20:31:09.316474 9849 tablet_bootstrap.cc:654] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:09.316489 9846 tablet_bootstrap.cc:492] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: No bootstrap required, opened a new log
I20260619 20:31:09.316563 9846 ts_tablet_manager.cc:1397] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260619 20:31:09.316819 9849 log.cc:824] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:09.317504 9849 tablet_bootstrap.cc:492] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: No bootstrap required, opened a new log
I20260619 20:31:09.317585 9849 ts_tablet_manager.cc:1397] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 20:31:09.318714 9846 raft_consensus.cc:348] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 } } peers { permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 } } peers { permanent_uuid: "8de38b8fe705407b87ba8f58242b591e" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42123 } }
I20260619 20:31:09.318902 9846 raft_consensus.cc:374] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:09.318945 9846 raft_consensus.cc:729] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 959bc7bf651c45278e47960f2465715a, State: Initialized, Role: FOLLOWER
I20260619 20:31:09.319082 9846 consensus_queue.cc:260] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 } } peers { permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 } } peers { permanent_uuid: "8de38b8fe705407b87ba8f58242b591e" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42123 } }
I20260619 20:31:09.319396 9846 ts_tablet_manager.cc:1428] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 20:31:09.319415 9700 heartbeater.cc:503] Master 127.21.151.62:39711 was elected leader, sending a full tablet report...
I20260619 20:31:09.319931 9849 raft_consensus.cc:348] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 } } peers { permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 } } peers { permanent_uuid: "8de38b8fe705407b87ba8f58242b591e" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42123 } }
I20260619 20:31:09.320094 9849 raft_consensus.cc:374] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:09.320140 9849 raft_consensus.cc:729] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcccf65019644ed98dae498ede7503f, State: Initialized, Role: FOLLOWER
I20260619 20:31:09.320269 9849 consensus_queue.cc:260] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 } } peers { permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 } } peers { permanent_uuid: "8de38b8fe705407b87ba8f58242b591e" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42123 } }
I20260619 20:31:09.320576 9849 ts_tablet_manager.cc:1428] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 20:31:09.320602 9831 heartbeater.cc:503] Master 127.21.151.62:39711 was elected leader, sending a full tablet report...
I20260619 20:31:09.358355 9850 raft_consensus.cc:482] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 20:31:09.358511 9850 raft_consensus.cc:504] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 } } peers { permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 } } peers { permanent_uuid: "8de38b8fe705407b87ba8f58242b591e" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42123 } }
I20260619 20:31:09.358901 9850 leader_election.cc:288] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcccf65019644ed98dae498ede7503f (127.21.151.3:42639), 959bc7bf651c45278e47960f2465715a (127.21.151.2:40575)
I20260619 20:31:09.362695 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.359158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:57219 (local address 127.21.151.3:42639)
0619 20:31:09.359202 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:09.359205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:09.359214 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:31:09.359247 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:09.359248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:09.359258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:09.359334 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:09.359488 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.360281 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.361746 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.362317 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.362392 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.362400 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:09.362588 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 20:31:09.362626 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 20:31:09.362695 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.359170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.151.1:48219 (local address 127.21.151.2:40575)
0619 20:31:09.359219 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:09.359221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:09.359367 (+ 146us) server_negotiation.cc:407] Connection header received
0619 20:31:09.359460 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:09.359461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:09.359469 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:09.359540 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 20:31:09.359733 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.360413 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.361744 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.362317 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.362392 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.362400 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:09.362588 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 20:31:09.362626 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 20:31:09.362701 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.359076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.3:42639 (local address 127.21.151.1:57219)
0619 20:31:09.359135 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:09.359147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:09.359188 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:09.359339 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:09.359340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:09.359341 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:09.359417 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.359419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.360398 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.360399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.361654 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.361657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.362401 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.362402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.362479 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.362486 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 20:31:09.362562 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 20:31:09.362607 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":12}
I20260619 20:31:09.362707 9854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.359116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.2:40575 (local address 127.21.151.1:48219)
0619 20:31:09.359290 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:09.359298 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:09.359384 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:09.359543 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:09.359543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:09.359545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 20:31:09.359664 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.359666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.360507 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.360508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.361655 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:09.361657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.362413 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.362414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:09.362479 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.362487 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 20:31:09.362562 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:31:09.362606 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":9,"thread_start_us":100,"threads_started":1}
I20260619 20:31:09.363194 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3b9bf7f3b040268cb1359fd18aaee4" candidate_uuid: "8de38b8fe705407b87ba8f58242b591e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959bc7bf651c45278e47960f2465715a" is_pre_election: true
I20260619 20:31:09.363260 9785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3b9bf7f3b040268cb1359fd18aaee4" candidate_uuid: "8de38b8fe705407b87ba8f58242b591e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcccf65019644ed98dae498ede7503f" is_pre_election: true
I20260619 20:31:09.363355 9654 raft_consensus.cc:2454] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de38b8fe705407b87ba8f58242b591e in term 0.
I20260619 20:31:09.363452 9785 raft_consensus.cc:2454] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8de38b8fe705407b87ba8f58242b591e in term 0.
I20260619 20:31:09.363582 9455 leader_election.cc:302] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8de38b8fe705407b87ba8f58242b591e, 959bc7bf651c45278e47960f2465715a; no voters:
I20260619 20:31:09.363749 9850 raft_consensus.cc:2781] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 20:31:09.363809 9850 raft_consensus.cc:482] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 20:31:09.363826 9850 raft_consensus.cc:3037] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:09.364544 9850 raft_consensus.cc:504] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 } } peers { permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 } } peers { permanent_uuid: "8de38b8fe705407b87ba8f58242b591e" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42123 } }
I20260619 20:31:09.364696 9850 leader_election.cc:288] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [CANDIDATE]: Term 1 election: Requested vote from peers dfcccf65019644ed98dae498ede7503f (127.21.151.3:42639), 959bc7bf651c45278e47960f2465715a (127.21.151.2:40575)
I20260619 20:31:09.364840 9785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3b9bf7f3b040268cb1359fd18aaee4" candidate_uuid: "8de38b8fe705407b87ba8f58242b591e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcccf65019644ed98dae498ede7503f"
I20260619 20:31:09.364898 9654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3b9bf7f3b040268cb1359fd18aaee4" candidate_uuid: "8de38b8fe705407b87ba8f58242b591e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959bc7bf651c45278e47960f2465715a"
I20260619 20:31:09.364943 9785 raft_consensus.cc:3037] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:09.365010 9654 raft_consensus.cc:3037] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:09.365682 9785 raft_consensus.cc:2454] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de38b8fe705407b87ba8f58242b591e in term 1.
I20260619 20:31:09.365850 9458 leader_election.cc:302] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8de38b8fe705407b87ba8f58242b591e, dfcccf65019644ed98dae498ede7503f; no voters:
I20260619 20:31:09.365880 9654 raft_consensus.cc:2454] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8de38b8fe705407b87ba8f58242b591e in term 1.
I20260619 20:31:09.365959 9850 raft_consensus.cc:2781] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:09.366130 9850 raft_consensus.cc:686] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 1 LEADER]: Becoming Leader. State: Replica: 8de38b8fe705407b87ba8f58242b591e, State: Running, Role: LEADER
I20260619 20:31:09.366235 9850 consensus_queue.cc:237] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 } } peers { permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 } } peers { permanent_uuid: "8de38b8fe705407b87ba8f58242b591e" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42123 } }
I20260619 20:31:09.367044 9380 catalog_manager.cc:5495] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8de38b8fe705407b87ba8f58242b591e (127.21.151.1). New cstate: current_term: 1 leader_uuid: "8de38b8fe705407b87ba8f58242b591e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de38b8fe705407b87ba8f58242b591e" member_type: VOTER last_known_addr { host: "127.21.151.1" port: 42123 } health_report { overall_health: HEALTHY } } }
I20260619 20:31:09.380033 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.377555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.21.151.1:42123)
0619 20:31:09.377599 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:09.377601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:09.377610 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:31:09.377666 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:09.377667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:09.377686 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:09.377773 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:09.377899 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.378380 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.379163 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:09.379376 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:09.379469 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:09.379669 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:09.379792 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:09.379888 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:09.379907 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 20:31:09.379934 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:31:09.381764 9654 raft_consensus.cc:1261] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a [term 1 FOLLOWER]: Refusing update from remote peer 8de38b8fe705407b87ba8f58242b591e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:09.381801 9785 raft_consensus.cc:1261] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f [term 1 FOLLOWER]: Refusing update from remote peer 8de38b8fe705407b87ba8f58242b591e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:09.382143 9855 consensus_queue.cc:1035] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [LEADER]: Connected to new peer: Peer: permanent_uuid: "959bc7bf651c45278e47960f2465715a" member_type: VOTER last_known_addr { host: "127.21.151.2" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:09.382274 9850 consensus_queue.cc:1035] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcccf65019644ed98dae498ede7503f" member_type: VOTER last_known_addr { host: "127.21.151.3" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:09.386749 9859 mvcc.cc:205] Tried to move back new op lower bound from 7298666780184981504 to 7298666780124803072. Current Snapshot: MvccSnapshot[applied={T|T < 7298666780184981504 or (T in {7298666780184981504})}]
I20260619 20:31:09.386931 9380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:09.387022 9380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 20:31:09.387766 9380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d336dee8eb2945778862dc136023a818: Sending DeleteTablet for 3 replicas of tablet 7b3b9bf7f3b040268cb1359fd18aaee4
I20260619 20:31:09.388103 9502 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3b9bf7f3b040268cb1359fd18aaee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:09 UTC) from {username='slave'} at 127.0.0.1:48354
I20260619 20:31:09.388120 9634 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3b9bf7f3b040268cb1359fd18aaee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:09 UTC) from {username='slave'} at 127.0.0.1:50882
I20260619 20:31:09.388134 9765 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3b9bf7f3b040268cb1359fd18aaee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 20:31:09 UTC) from {username='slave'} at 127.0.0.1:58258
I20260619 20:31:09.388379 9870 tablet_replica.cc:330] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e: stopping tablet replica
I20260619 20:31:09.388501 9869 tablet_replica.cc:330] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: stopping tablet replica
I20260619 20:31:09.388583 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 9440
I20260619 20:31:09.388728 9870 raft_consensus.cc:2229] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 8de38b8fe705407b87ba8f58242b591e [term 1 LEADER]: Raft consensus shutting down.
I20260619 20:31:09.389427 9869 raft_consensus.cc:2229] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:31:09.389616 9869 raft_consensus.cc:2258] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:09.389837 9868 tablet_replica.cc:330] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: stopping tablet replica
I20260619 20:31:09.389962 9868 raft_consensus.cc:2229] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 20:31:09.390100 9868 raft_consensus.cc:2258] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 20:31:09.390146 9869 ts_tablet_manager.cc:1905] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:09.390607 9868 ts_tablet_manager.cc:1905] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 20:31:09.391764 9869 ts_tablet_manager.cc:1918] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:09.391853 9869 log.cc:1196] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-2/wal/wals/7b3b9bf7f3b040268cb1359fd18aaee4
I20260619 20:31:09.392103 9869 ts_tablet_manager.cc:1939] T 7b3b9bf7f3b040268cb1359fd18aaee4 P dfcccf65019644ed98dae498ede7503f: Deleting consensus metadata
I20260619 20:31:09.392531 9367 catalog_manager.cc:4841] TS dfcccf65019644ed98dae498ede7503f (127.21.151.3:42639): tablet 7b3b9bf7f3b040268cb1359fd18aaee4 (table test-table [id=4972824f5a99429fa4bdde4267bab108]) successfully deleted
I20260619 20:31:09.392617 9868 ts_tablet_manager.cc:1918] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 20:31:09.392691 9868 log.cc:1196] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: Deleting WAL directory at /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781900829246020-22108-0/minicluster-data/ts-1/wal/wals/7b3b9bf7f3b040268cb1359fd18aaee4
I20260619 20:31:09.392916 9868 ts_tablet_manager.cc:1939] T 7b3b9bf7f3b040268cb1359fd18aaee4 P 959bc7bf651c45278e47960f2465715a: Deleting consensus metadata
I20260619 20:31:09.393325 9364 catalog_manager.cc:4841] TS 959bc7bf651c45278e47960f2465715a (127.21.151.2:40575): tablet 7b3b9bf7f3b040268cb1359fd18aaee4 (table test-table [id=4972824f5a99429fa4bdde4267bab108]) successfully deleted
W20260619 20:31:09.394970 9367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.151.1:42123 (error 108)
I20260619 20:31:09.395226 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 9571
I20260619 20:31:09.395828 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:09.395274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.1:42123 (local address 127.0.0.1:48380)
0619 20:31:09.395745 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:09.395788 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.1:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450}
W20260619 20:31:09.396023 9367 catalog_manager.cc:4584] TS 8de38b8fe705407b87ba8f58242b591e (127.21.151.1:42123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b3b9bf7f3b040268cb1359fd18aaee4: Network error: Client connection negotiation failed: client connection to 127.21.151.1:42123: connect: Connection refused (error 111)
I20260619 20:31:09.399920 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 9703
I20260619 20:31:09.406348 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 9349
2026-06-19T20:31:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (680 ms)
[----------] 6 tests from EncryptionPolicyTest (8508 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:09 dist-test-slave-f9rk krb5kdc[9874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:09 dist-test-slave-f9rk krb5kdc[9874](info): set up 2 sockets
Jun 19 20:31:09 dist-test-slave-f9rk krb5kdc[9874](info): commencing operation
krb5kdc: starting...
W20260619 20:31:10.440694 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:10 dist-test-slave-f9rk krb5kdc[9874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:10.555809 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:10.634575 9887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:10.634810 9887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:10.634855 9887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:10.636023 9887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:10.636086 9887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:10.636109 9887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:10.636129 9887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:10.636154 9887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:10.637534 9887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:10.637789 9887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:10.638010 9887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:10.642786 9893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:10.642751 9895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:10.642810 9892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:10.642922 9887 server_base.cc:1034] running on GCE node
I20260619 20:31:10.643157 9887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:10.643441 9887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:10.644603 9887 hybrid_clock.cc:648] HybridClock initialized: now 1781901070644567 us; error 57 us; skew 500 ppm
Jun 19 20:31:10 dist-test-slave-f9rk krb5kdc[9874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:10.649935 9887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:10.650395 9887 webserver.cc:460] Webserver started at http://127.0.0.1:37635/ using document root <none> and password file <none>
I20260619 20:31:10.650633 9887 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:10.650678 9887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:10.650812 9887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:10.651971 9887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "e85ba67d28574a0d98a96a5995355b73"
format_stamp: "Formatted at 2026-06-19 20:31:10 on dist-test-slave-f9rk"
server_key: "5371a8e55f4ba1e927bdd33fa8b53d28"
server_key_iv: "0ef6c8bbeaf57cf6ad2a4441cd09f7ce"
server_key_version: "encryptionkey@0"
I20260619 20:31:10.652370 9887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "e85ba67d28574a0d98a96a5995355b73"
format_stamp: "Formatted at 2026-06-19 20:31:10 on dist-test-slave-f9rk"
server_key: "5371a8e55f4ba1e927bdd33fa8b53d28"
server_key_iv: "0ef6c8bbeaf57cf6ad2a4441cd09f7ce"
server_key_version: "encryptionkey@0"
I20260619 20:31:10.653689 9887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:10.654421 9902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:10.654604 9887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:10.654711 9887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "e85ba67d28574a0d98a96a5995355b73"
format_stamp: "Formatted at 2026-06-19 20:31:10 on dist-test-slave-f9rk"
server_key: "5371a8e55f4ba1e927bdd33fa8b53d28"
server_key_iv: "0ef6c8bbeaf57cf6ad2a4441cd09f7ce"
server_key_version: "encryptionkey@0"
I20260619 20:31:10.654769 9887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:10.678539 9887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:10.680975 9887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:10.681088 9887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:10.684912 9887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36281
I20260619 20:31:10.684978 9954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36281 every 8 connection(s)
I20260619 20:31:10.685323 9887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:10.686225 9955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:10.688598 9955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73: Bootstrap starting.
I20260619 20:31:10.689105 9955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:10.689321 9955 log.cc:824] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:10.689975 9955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73: No bootstrap required, opened a new log
I20260619 20:31:10.690673 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 9887
I20260619 20:31:10.690774 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:10.690995 22108 external_mini_cluster.cc:1428] Setting key 795b82cf75618bc30d97f915829f1702
I20260619 20:31:10.691972 9955 raft_consensus.cc:348] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85ba67d28574a0d98a96a5995355b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260619 20:31:10.692159 9955 raft_consensus.cc:374] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:10.692204 9955 raft_consensus.cc:729] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85ba67d28574a0d98a96a5995355b73, State: Initialized, Role: FOLLOWER
I20260619 20:31:10.692340 9955 consensus_queue.cc:260] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85ba67d28574a0d98a96a5995355b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260619 20:31:10.692431 9955 raft_consensus.cc:388] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:10.692466 9955 raft_consensus.cc:482] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:10.692516 9955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:10.693473 9955 raft_consensus.cc:504] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85ba67d28574a0d98a96a5995355b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260619 20:31:10.693558 9955 leader_election.cc:302] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e85ba67d28574a0d98a96a5995355b73; no voters:
I20260619 20:31:10.693737 9955 leader_election.cc:288] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:10.693797 9960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:10.693938 9960 raft_consensus.cc:686] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [term 1 LEADER]: Becoming Leader. State: Replica: e85ba67d28574a0d98a96a5995355b73, State: Running, Role: LEADER
I20260619 20:31:10.693964 9955 sys_catalog.cc:564] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:10.694188 9960 consensus_queue.cc:237] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85ba67d28574a0d98a96a5995355b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } }
I20260619 20:31:10.694581 9962 sys_catalog.cc:455] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e85ba67d28574a0d98a96a5995355b73. Latest consensus state: current_term: 1 leader_uuid: "e85ba67d28574a0d98a96a5995355b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85ba67d28574a0d98a96a5995355b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } } }
I20260619 20:31:10.694677 9962 sys_catalog.cc:458] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:10.694902 9969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:10.695416 9969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:10.695737 9961 sys_catalog.cc:455] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e85ba67d28574a0d98a96a5995355b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85ba67d28574a0d98a96a5995355b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36281 } } }
I20260619 20:31:10.695825 9961 sys_catalog.cc:458] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:10.697213 9969 catalog_manager.cc:1347] Generated new cluster ID: 633d9a1675544d8bad511bef0812baff
I20260619 20:31:10.697265 9969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:31:10 dist-test-slave-f9rk krb5kdc[9874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:31:10.702744 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:10.691527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56927 (local address 127.0.0.1:36281)
0619 20:31:10.691705 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:10.691713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:10.691738 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:31:10.691951 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:10.691957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:10.692000 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:10.692110 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:10.692927 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:10.694047 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:10.695208 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:10.695895 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:10.696032 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:10.698876 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:10.698880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:10.698880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:10.698891 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:10.700877 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:10.701295 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:10.701298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:10.701298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:10.701429 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:10.701662 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:10.701663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:10.701663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:10.701991 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:10.702174 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:10.702253 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 20:31:10.702369 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260619 20:31:10.703775 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:10.712051 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:10.705191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45251 (local address 127.0.0.1:36281)
0619 20:31:10.705227 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:10.705228 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:10.705400 (+ 172us) server_negotiation.cc:407] Connection header received
0619 20:31:10.705444 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:10.705445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:10.705455 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:10.705520 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:10.706091 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:10.706808 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:10.707774 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:10.708063 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:10.708096 (+ 33us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:10.709379 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:10.709380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:10.709380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:10.709386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:10.710885 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:10.711263 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:10.711264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:10.711264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:10.711309 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:10.711498 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:10.711499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:10.711499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:10.711613 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:10.711714 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:10.711909 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 20:31:10.711965 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:31:10.718103 9969 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:10.718796 9969 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:10.724830 9969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e85ba67d28574a0d98a96a5995355b73: Generated new TSK 0
I20260619 20:31:10.724996 9969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:10.731351 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 9887
W20260619 20:31:10.735680 9872 connection.cc:535] client connection to 127.0.0.1:36281 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36281): Connection reset by peer (error 104)
2026-06-19T20:31:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:10 dist-test-slave-f9rk krb5kdc[9986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:10 dist-test-slave-f9rk krb5kdc[9986](info): set up 2 sockets
Jun 19 20:31:10 dist-test-slave-f9rk krb5kdc[9986](info): commencing operation
krb5kdc: starting...
W20260619 20:31:11.764940 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:11 dist-test-slave-f9rk krb5kdc[9986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:11.879849 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:11.959455 9999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:11.959717 9999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:11.959744 9999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:11.960945 9999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:11.960987 9999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:11.961001 9999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:11.961014 9999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:11.961026 9999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:11.962563 9999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:11.962811 9999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:11.963038 9999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:11.967962 10007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:11.967990 10005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:11.968118 9999 server_base.cc:1034] running on GCE node
W20260619 20:31:11.968071 10004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:11.968585 9999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:11.968880 9999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:11.970065 9999 hybrid_clock.cc:648] HybridClock initialized: now 1781901071970049 us; error 45 us; skew 500 ppm
Jun 19 20:31:11 dist-test-slave-f9rk krb5kdc[9986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:11.972488 9999 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:11.972960 9999 webserver.cc:460] Webserver started at http://127.0.0.1:39301/ using document root <none> and password file <none>
I20260619 20:31:11.973182 9999 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:11.973248 9999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:11.973392 9999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:11.974427 9999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "77f94c693fe24d1da57aac678c1d64a7"
format_stamp: "Formatted at 2026-06-19 20:31:11 on dist-test-slave-f9rk"
server_key: "622387cf978ea9fc7b0ec16ee2228b1b"
server_key_iv: "3f4e2bd1588dbd062af9681566e47ac7"
server_key_version: "encryptionkey@0"
I20260619 20:31:11.974776 9999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "77f94c693fe24d1da57aac678c1d64a7"
format_stamp: "Formatted at 2026-06-19 20:31:11 on dist-test-slave-f9rk"
server_key: "622387cf978ea9fc7b0ec16ee2228b1b"
server_key_iv: "3f4e2bd1588dbd062af9681566e47ac7"
server_key_version: "encryptionkey@0"
I20260619 20:31:11.976114 9999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:31:11.976830 10014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:11.977011 9999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:11.977118 9999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "77f94c693fe24d1da57aac678c1d64a7"
format_stamp: "Formatted at 2026-06-19 20:31:11 on dist-test-slave-f9rk"
server_key: "622387cf978ea9fc7b0ec16ee2228b1b"
server_key_iv: "3f4e2bd1588dbd062af9681566e47ac7"
server_key_version: "encryptionkey@0"
I20260619 20:31:11.977195 9999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:11.987258 9999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:11.989054 9999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:11.989223 9999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:11.993288 9999 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35157
I20260619 20:31:11.993319 10066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35157 every 8 connection(s)
I20260619 20:31:11.993664 9999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:11.994289 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 9999
I20260619 20:31:11.994405 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:11.994627 10067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:11.994632 22108 external_mini_cluster.cc:1428] Setting key 4809ade5bda483d65124eb44c808a131
I20260619 20:31:11.997987 10067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7: Bootstrap starting.
I20260619 20:31:11.998579 10067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:11.998872 10067 log.cc:824] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:11.999600 10067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7: No bootstrap required, opened a new log
I20260619 20:31:12.000222 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:11.995187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51575 (local address 127.0.0.1:35157)
0619 20:31:11.995377 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:11.995385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:11.995409 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:31:11.995857 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:11.995864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:11.995914 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:11.996027 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:11.996270 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:11.997140 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:11.998208 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:11.998610 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:11.998733 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:11.999506 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:11.999510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:11.999510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:11.999745 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:11.999787 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:11.999802 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:11.999891 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:11.999982 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 20:31:12.000008 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260619 20:31:12.001485 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:12.001958 10067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f94c693fe24d1da57aac678c1d64a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260619 20:31:12.002091 10067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:12.002115 10067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f94c693fe24d1da57aac678c1d64a7, State: Initialized, Role: FOLLOWER
I20260619 20:31:12.002249 10067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f94c693fe24d1da57aac678c1d64a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260619 20:31:12.002326 10067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:12.002347 10067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:12.002422 10067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:12.003031 10067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f94c693fe24d1da57aac678c1d64a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260619 20:31:12.003168 10067 leader_election.cc:302] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77f94c693fe24d1da57aac678c1d64a7; no voters:
I20260619 20:31:12.003337 10067 leader_election.cc:288] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:12.003384 10077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:12.003526 10077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [term 1 LEADER]: Becoming Leader. State: Replica: 77f94c693fe24d1da57aac678c1d64a7, State: Running, Role: LEADER
I20260619 20:31:12.003571 10067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:12.003639 10077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f94c693fe24d1da57aac678c1d64a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } }
I20260619 20:31:12.003996 10078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77f94c693fe24d1da57aac678c1d64a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f94c693fe24d1da57aac678c1d64a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } } }
I20260619 20:31:12.004112 10078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:12.003998 10079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77f94c693fe24d1da57aac678c1d64a7. Latest consensus state: current_term: 1 leader_uuid: "77f94c693fe24d1da57aac678c1d64a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77f94c693fe24d1da57aac678c1d64a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35157 } } }
I20260619 20:31:12.004381 10079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:12.004578 10084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:12.005087 10084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:12.006659 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:12.002881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39607 (local address 127.0.0.1:35157)
0619 20:31:12.002938 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:12.002940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:12.003088 (+ 148us) server_negotiation.cc:407] Connection header received
0619 20:31:12.003112 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:12.003112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:12.003123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:12.003174 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:12.003309 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:12.004049 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:12.005317 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:12.005556 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:12.005640 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:12.006327 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:12.006328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:12.006329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:12.006339 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:12.006354 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:12.006358 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:12.006417 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:12.006575 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 20:31:12.006585 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 20:31:12.007028 10084 catalog_manager.cc:1347] Generated new cluster ID: 9d4a422aa8f748ecb02d42aa19faf261
I20260619 20:31:12.007076 10084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:12.021191 10084 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:12.021890 10084 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:12.024504 10084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77f94c693fe24d1da57aac678c1d64a7: Generated new TSK 0
I20260619 20:31:12.024667 10084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:12.032083 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 9999
W20260619 20:31:12.036619 9984 connection.cc:535] client connection to 127.0.0.1:35157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35157): Connection reset by peer (error 104)
2026-06-19T20:31:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:12 dist-test-slave-f9rk krb5kdc[10098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:12 dist-test-slave-f9rk krb5kdc[10098](info): set up 2 sockets
Jun 19 20:31:12 dist-test-slave-f9rk krb5kdc[10098](info): commencing operation
krb5kdc: starting...
W20260619 20:31:13.065035 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:13 dist-test-slave-f9rk krb5kdc[10098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:13.185686 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:13.266597 10111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:13.266801 10111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:13.266824 10111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:13.268006 10111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:13.268046 10111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:13.268061 10111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:13.268074 10111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:13.268133 10111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:13.269644 10111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:13.269891 10111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:13.270123 10111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:13.274893 10117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:13.274907 10119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:13.274905 10116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:13.275346 10111 server_base.cc:1034] running on GCE node
I20260619 20:31:13.275594 10111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:13.275936 10111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:13.277081 10111 hybrid_clock.cc:648] HybridClock initialized: now 1781901073277061 us; error 38 us; skew 500 ppm
Jun 19 20:31:13 dist-test-slave-f9rk krb5kdc[10098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:13.279521 10111 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:13.280007 10111 webserver.cc:460] Webserver started at http://127.0.0.1:35563/ using document root <none> and password file <none>
I20260619 20:31:13.280203 10111 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:13.280253 10111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:13.280349 10111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:13.281369 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "afcf7d6144d04a598a11856cae02826e"
format_stamp: "Formatted at 2026-06-19 20:31:13 on dist-test-slave-f9rk"
server_key: "0ce5a97a5a1990c312aac2bce399f0ef"
server_key_iv: "170cdfa815f493ed7985affa54c15cf8"
server_key_version: "encryptionkey@0"
I20260619 20:31:13.281662 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "afcf7d6144d04a598a11856cae02826e"
format_stamp: "Formatted at 2026-06-19 20:31:13 on dist-test-slave-f9rk"
server_key: "0ce5a97a5a1990c312aac2bce399f0ef"
server_key_iv: "170cdfa815f493ed7985affa54c15cf8"
server_key_version: "encryptionkey@0"
I20260619 20:31:13.282876 10111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:31:13.283537 10126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:13.283725 10111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:13.283823 10111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "afcf7d6144d04a598a11856cae02826e"
format_stamp: "Formatted at 2026-06-19 20:31:13 on dist-test-slave-f9rk"
server_key: "0ce5a97a5a1990c312aac2bce399f0ef"
server_key_iv: "170cdfa815f493ed7985affa54c15cf8"
server_key_version: "encryptionkey@0"
I20260619 20:31:13.283905 10111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:13.287065 10111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:13.288938 10111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:13.289093 10111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:13.293020 10111 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43069
I20260619 20:31:13.293051 10178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43069 every 8 connection(s)
I20260619 20:31:13.293385 10111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:13.294303 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:13.296787 10179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e: Bootstrap starting.
I20260619 20:31:13.297359 10179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:13.297653 10179 log.cc:824] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:13.298306 10179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e: No bootstrap required, opened a new log
I20260619 20:31:13.299963 10179 raft_consensus.cc:348] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcf7d6144d04a598a11856cae02826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43069 } }
I20260619 20:31:13.300077 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 10111
I20260619 20:31:13.300120 10179 raft_consensus.cc:374] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:13.300187 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:13.300168 10179 raft_consensus.cc:729] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afcf7d6144d04a598a11856cae02826e, State: Initialized, Role: FOLLOWER
I20260619 20:31:13.300376 22108 external_mini_cluster.cc:1428] Setting key 26cf83507033bae93880e896c9b3dac5
I20260619 20:31:13.300356 10179 consensus_queue.cc:260] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcf7d6144d04a598a11856cae02826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43069 } }
I20260619 20:31:13.300459 10179 raft_consensus.cc:388] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:13.300513 10179 raft_consensus.cc:482] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:13.300570 10179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:13.301244 10179 raft_consensus.cc:504] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcf7d6144d04a598a11856cae02826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43069 } }
I20260619 20:31:13.301366 10179 leader_election.cc:302] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afcf7d6144d04a598a11856cae02826e; no voters:
I20260619 20:31:13.301561 10179 leader_election.cc:288] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:13.301579 10184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:13.301699 10184 raft_consensus.cc:686] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [term 1 LEADER]: Becoming Leader. State: Replica: afcf7d6144d04a598a11856cae02826e, State: Running, Role: LEADER
I20260619 20:31:13.301819 10179 sys_catalog.cc:564] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:13.301877 10184 consensus_queue.cc:237] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcf7d6144d04a598a11856cae02826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43069 } }
I20260619 20:31:13.302271 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [sys.catalog]: SysCatalogTable state changed. Reason: New leader afcf7d6144d04a598a11856cae02826e. Latest consensus state: current_term: 1 leader_uuid: "afcf7d6144d04a598a11856cae02826e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcf7d6144d04a598a11856cae02826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43069 } } }
I20260619 20:31:13.302282 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afcf7d6144d04a598a11856cae02826e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcf7d6144d04a598a11856cae02826e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43069 } } }
I20260619 20:31:13.302385 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:13.302491 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:13.302721 10188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:13.303365 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:13.300846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34521 (local address 127.0.0.1:43069)
0619 20:31:13.301348 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:13.301374 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:13.301406 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:31:13.301533 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:13.301539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:13.301571 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:13.301716 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:13.302783 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:13.302787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:13.302787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:13.302824 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:13.302874 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:13.302890 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:13.302951 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:13.303079 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 20:31:13.303121 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":412,"threads_started":1}
I20260619 20:31:13.303911 10188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:13.304436 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:13.305717 10188 catalog_manager.cc:1347] Generated new cluster ID: 3ce9241224b047708a17f7045e0a74d4
I20260619 20:31:13.305768 10188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:13.308799 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:13.305965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57607 (local address 127.0.0.1:43069)
0619 20:31:13.306003 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:13.306005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:13.307869 (+ 1864us) server_negotiation.cc:407] Connection header received
0619 20:31:13.307931 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:13.307932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:13.307937 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:13.307995 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:13.308596 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:13.308597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:13.308598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:13.308602 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:13.308615 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:13.308619 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:13.308669 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:13.308727 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 20:31:13.308739 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:31:13.322964 10188 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:13.323153 10188 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:13.326318 10188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afcf7d6144d04a598a11856cae02826e: Generated new TSK 0
I20260619 20:31:13.326526 10188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:13.335112 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 10111
2026-06-19T20:31:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:13 dist-test-slave-f9rk krb5kdc[10210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:13 dist-test-slave-f9rk krb5kdc[10210](info): set up 2 sockets
Jun 19 20:31:13 dist-test-slave-f9rk krb5kdc[10210](info): commencing operation
krb5kdc: starting...
W20260619 20:31:14.360808 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:14 dist-test-slave-f9rk krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:14.474337 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:14.553421 10223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:14.553665 10223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:14.553701 10223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:14.554840 10223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:14.554911 10223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:14.554937 10223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:14.554965 10223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:14.554989 10223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:14.556527 10223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:14.556790 10223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:14.557029 10223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:14.561822 10228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:14.561818 10231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:14.561849 10229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:14.562171 10223 server_base.cc:1034] running on GCE node
I20260619 20:31:14.562458 10223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:14.562736 10223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:14.563874 10223 hybrid_clock.cc:648] HybridClock initialized: now 1781901074563867 us; error 30 us; skew 500 ppm
Jun 19 20:31:14 dist-test-slave-f9rk krb5kdc[10210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:14.566237 10223 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:14.566664 10223 webserver.cc:460] Webserver started at http://127.0.0.1:46757/ using document root <none> and password file <none>
I20260619 20:31:14.566917 10223 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:14.566964 10223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:14.567129 10223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:14.568255 10223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "b5af4093c8ef498aa62154e86dfe78c1"
format_stamp: "Formatted at 2026-06-19 20:31:14 on dist-test-slave-f9rk"
server_key: "afb06c19099dd0c97f1214903d12e89a"
server_key_iv: "5ae20e75f808b2d333d8a74a1ce7f831"
server_key_version: "encryptionkey@0"
I20260619 20:31:14.568658 10223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "b5af4093c8ef498aa62154e86dfe78c1"
format_stamp: "Formatted at 2026-06-19 20:31:14 on dist-test-slave-f9rk"
server_key: "afb06c19099dd0c97f1214903d12e89a"
server_key_iv: "5ae20e75f808b2d333d8a74a1ce7f831"
server_key_version: "encryptionkey@0"
I20260619 20:31:14.569952 10223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:14.570641 10238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:14.570809 10223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:14.570919 10223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "b5af4093c8ef498aa62154e86dfe78c1"
format_stamp: "Formatted at 2026-06-19 20:31:14 on dist-test-slave-f9rk"
server_key: "afb06c19099dd0c97f1214903d12e89a"
server_key_iv: "5ae20e75f808b2d333d8a74a1ce7f831"
server_key_version: "encryptionkey@0"
I20260619 20:31:14.570993 10223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:14.587157 10223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:14.589084 10223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:14.589267 10223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:14.593094 10223 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39719
I20260619 20:31:14.593150 10290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39719 every 8 connection(s)
I20260619 20:31:14.593520 10223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:14.594379 10291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:14.596602 10291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1: Bootstrap starting.
I20260619 20:31:14.597167 10291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:14.597453 10291 log.cc:824] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:14.598121 10291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1: No bootstrap required, opened a new log
I20260619 20:31:14.599125 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 10223
I20260619 20:31:14.599277 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:14.599488 22108 external_mini_cluster.cc:1428] Setting key 859a463323b7fae355383eba1738c2b0
I20260619 20:31:14.600268 10291 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5af4093c8ef498aa62154e86dfe78c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260619 20:31:14.600457 10291 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:14.600493 10291 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5af4093c8ef498aa62154e86dfe78c1, State: Initialized, Role: FOLLOWER
I20260619 20:31:14.600649 10291 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5af4093c8ef498aa62154e86dfe78c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260619 20:31:14.600733 10291 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:14.600775 10291 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:14.600827 10291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:14.601871 10291 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5af4093c8ef498aa62154e86dfe78c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260619 20:31:14.601971 10291 leader_election.cc:302] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5af4093c8ef498aa62154e86dfe78c1; no voters:
I20260619 20:31:14.602130 10291 leader_election.cc:288] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:14.602208 10296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:14.602353 10291 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:14.602385 10296 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [term 1 LEADER]: Becoming Leader. State: Replica: b5af4093c8ef498aa62154e86dfe78c1, State: Running, Role: LEADER
I20260619 20:31:14.602566 10296 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5af4093c8ef498aa62154e86dfe78c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } }
I20260619 20:31:14.603001 10297 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5af4093c8ef498aa62154e86dfe78c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5af4093c8ef498aa62154e86dfe78c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } } }
I20260619 20:31:14.603094 10297 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:14.602983 10298 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5af4093c8ef498aa62154e86dfe78c1. Latest consensus state: current_term: 1 leader_uuid: "b5af4093c8ef498aa62154e86dfe78c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5af4093c8ef498aa62154e86dfe78c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39719 } } }
I20260619 20:31:14.603317 10305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:14.603372 10298 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:14.604008 10305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:14.605624 10305 catalog_manager.cc:1347] Generated new cluster ID: 86ca409c5d184b15b630c9e9717ed4c7
I20260619 20:31:14.606254 10305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:31:14 dist-test-slave-f9rk krb5kdc[10210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:31:14.610787 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:14.600023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37735 (local address 127.0.0.1:39719)
0619 20:31:14.600235 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:14.600244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:14.600278 (+ 34us) server_negotiation.cc:407] Connection header received
0619 20:31:14.600397 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:14.600404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:14.600477 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:14.600623 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:14.601324 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:14.602748 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:14.603889 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:14.604376 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:14.604518 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:14.607177 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:14.607181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:14.607181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:14.607193 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:14.609150 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:14.609563 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:14.609566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:14.609566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:14.609687 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:14.609900 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:14.609901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:14.609901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:14.610177 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:14.610331 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:14.610471 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 20:31:14.610593 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
I20260619 20:31:14.611840 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:14.620039 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:14.613232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59001 (local address 127.0.0.1:39719)
0619 20:31:14.613261 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:14.613263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:14.613383 (+ 120us) server_negotiation.cc:407] Connection header received
0619 20:31:14.613441 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:14.613442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:14.613461 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:14.613499 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:14.614134 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:14.614842 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:14.615855 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:14.616186 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:14.616248 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:14.617506 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:14.617507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:14.617508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:14.617513 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:14.618983 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:14.619338 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:14.619339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:14.619339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:14.619388 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:14.619570 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:14.619571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:14.619571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:14.619689 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:14.619788 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:14.619898 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 20:31:14.619962 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 20:31:14.623040 10305 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:14.623564 10305 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:14.630728 10305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5af4093c8ef498aa62154e86dfe78c1: Generated new TSK 0
I20260619 20:31:14.630923 10305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:14.638058 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 10223
2026-06-19T20:31:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:14 dist-test-slave-f9rk krb5kdc[10322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:14 dist-test-slave-f9rk krb5kdc[10322](info): set up 2 sockets
Jun 19 20:31:14 dist-test-slave-f9rk krb5kdc[10322](info): commencing operation
krb5kdc: starting...
W20260619 20:31:15.668610 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:15 dist-test-slave-f9rk krb5kdc[10322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:15.784107 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:15.862075 10335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:15.862287 10335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:15.862324 10335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:15.863385 10335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:15.863451 10335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:15.863476 10335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:15.863497 10335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:15.863523 10335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:15.864923 10335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:15.865182 10335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:15.865398 10335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:31:15.870040 10335 server_base.cc:1034] running on GCE node
W20260619 20:31:15.870024 10340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:15.870045 10341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:15.870025 10343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:15.870404 10335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:15.870676 10335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:15.871812 10335 hybrid_clock.cc:648] HybridClock initialized: now 1781901075871807 us; error 44 us; skew 500 ppm
Jun 19 20:31:15 dist-test-slave-f9rk krb5kdc[10322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:15.876797 10335 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:15.877247 10335 webserver.cc:460] Webserver started at http://127.0.0.1:33943/ using document root <none> and password file <none>
I20260619 20:31:15.877490 10335 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:15.877533 10335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:15.877733 10335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:15.878732 10335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "e9375c9edd5c4f38890d23dbb72c61e1"
format_stamp: "Formatted at 2026-06-19 20:31:15 on dist-test-slave-f9rk"
server_key: "f716fb6a8e036ead7517e06c0b0330be"
server_key_iv: "4e34aa52497305906d900bc50663980b"
server_key_version: "encryptionkey@0"
I20260619 20:31:15.879081 10335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "e9375c9edd5c4f38890d23dbb72c61e1"
format_stamp: "Formatted at 2026-06-19 20:31:15 on dist-test-slave-f9rk"
server_key: "f716fb6a8e036ead7517e06c0b0330be"
server_key_iv: "4e34aa52497305906d900bc50663980b"
server_key_version: "encryptionkey@0"
I20260619 20:31:15.880438 10335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:31:15.881179 10350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:15.881366 10335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:15.881453 10335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "e9375c9edd5c4f38890d23dbb72c61e1"
format_stamp: "Formatted at 2026-06-19 20:31:15 on dist-test-slave-f9rk"
server_key: "f716fb6a8e036ead7517e06c0b0330be"
server_key_iv: "4e34aa52497305906d900bc50663980b"
server_key_version: "encryptionkey@0"
I20260619 20:31:15.881524 10335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:15.898612 10335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:15.900509 10335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:15.900667 10335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:15.904729 10335 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37581
I20260619 20:31:15.904767 10402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37581 every 8 connection(s)
I20260619 20:31:15.905148 10335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:15.905995 10403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:15.908352 10403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1: Bootstrap starting.
I20260619 20:31:15.908885 10403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:15.909114 10403 log.cc:824] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:15.909461 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 10335
I20260619 20:31:15.909611 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:15.909806 22108 external_mini_cluster.cc:1428] Setting key dd3cd140a42944875f3dca4621291a94
I20260619 20:31:15.909996 10403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1: No bootstrap required, opened a new log
I20260619 20:31:15.912953 10403 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9375c9edd5c4f38890d23dbb72c61e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37581 } }
I20260619 20:31:15.913122 10403 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:15.913152 10403 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9375c9edd5c4f38890d23dbb72c61e1, State: Initialized, Role: FOLLOWER
I20260619 20:31:15.913264 10403 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9375c9edd5c4f38890d23dbb72c61e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37581 } }
I20260619 20:31:15.913353 10403 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:15.913391 10403 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:15.913452 10403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:15.913987 10403 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9375c9edd5c4f38890d23dbb72c61e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37581 } }
I20260619 20:31:15.914074 10403 leader_election.cc:302] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9375c9edd5c4f38890d23dbb72c61e1; no voters:
I20260619 20:31:15.914258 10403 leader_election.cc:288] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:15.914280 10408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:15.914392 10408 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [term 1 LEADER]: Becoming Leader. State: Replica: e9375c9edd5c4f38890d23dbb72c61e1, State: Running, Role: LEADER
I20260619 20:31:15.914526 10408 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9375c9edd5c4f38890d23dbb72c61e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37581 } }
I20260619 20:31:15.914528 10403 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:15.915747 10409 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9375c9edd5c4f38890d23dbb72c61e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9375c9edd5c4f38890d23dbb72c61e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37581 } } }
I20260619 20:31:15.915854 10409 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:15.916306 10410 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9375c9edd5c4f38890d23dbb72c61e1. Latest consensus state: current_term: 1 leader_uuid: "e9375c9edd5c4f38890d23dbb72c61e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9375c9edd5c4f38890d23dbb72c61e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37581 } } }
I20260619 20:31:15.916383 10410 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:15.916476 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:15.910334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46163 (local address 127.0.0.1:37581)
0619 20:31:15.910544 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:15.910554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:15.910589 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:31:15.911399 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:15.911409 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:15.911474 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:15.911647 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:15.911867 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:15.913301 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:15.914167 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:15.914757 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:15.914911 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:15.915665 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:15.915669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:15.915670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:15.915880 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:15.915919 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:15.915934 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:15.915994 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:15.916199 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 20:31:15.916263 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
W20260619 20:31:15.916623 10424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:31:15.916682 10424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:31:15.916937 10422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:15.917145 10422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:15.918102 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:15.918502 10422 catalog_manager.cc:1347] Generated new cluster ID: ac26ad2bdbdc4c76a63ee33bb6820715
I20260619 20:31:15.918548 10422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:15.923050 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:15.919631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47285 (local address 127.0.0.1:37581)
0619 20:31:15.919666 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:15.919667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:15.919722 (+ 55us) server_negotiation.cc:407] Connection header received
0619 20:31:15.919770 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:15.919771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:15.919780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:15.919852 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:15.919952 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:15.920703 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:15.921675 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:15.922005 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:15.922097 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:15.922818 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:15.922819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:15.922820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:15.922825 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:15.922840 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:15.922844 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:15.922921 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:15.922963 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 20:31:15.922982 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:31:15.936174 10422 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:15.936777 10422 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:15.945206 10422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9375c9edd5c4f38890d23dbb72c61e1: Generated new TSK 0
I20260619 20:31:15.945411 10422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:15.948297 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 10335
2026-06-19T20:31:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:15 dist-test-slave-f9rk krb5kdc[10434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:15 dist-test-slave-f9rk krb5kdc[10434](info): set up 2 sockets
Jun 19 20:31:15 dist-test-slave-f9rk krb5kdc[10434](info): commencing operation
krb5kdc: starting...
W20260619 20:31:16.972666 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:17 dist-test-slave-f9rk krb5kdc[10434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:17.089824 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:17.169981 10447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:17.170171 10447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:17.170193 10447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:17.171355 10447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:17.171396 10447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:17.171417 10447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:17.171432 10447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:17.171491 10447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:17.172914 10447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:17.173158 10447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:17.173377 10447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:17.178166 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:17.178146 10453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:17.178146 10452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:17.178162 10447 server_base.cc:1034] running on GCE node
I20260619 20:31:17.178602 10447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:17.178884 10447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:17.180049 10447 hybrid_clock.cc:648] HybridClock initialized: now 1781901077180043 us; error 35 us; skew 500 ppm
Jun 19 20:31:17 dist-test-slave-f9rk krb5kdc[10434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:17.182448 10447 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:17.182865 10447 webserver.cc:460] Webserver started at http://127.0.0.1:45095/ using document root <none> and password file <none>
I20260619 20:31:17.183084 10447 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:17.183151 10447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:17.183298 10447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:17.184425 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "d833ebb9b8b044b7a24a2ab1b09ba771"
format_stamp: "Formatted at 2026-06-19 20:31:17 on dist-test-slave-f9rk"
server_key: "9c73db9091e24c7474b406d5964582c9"
server_key_iv: "d2894c049fc97933e30ec3dee815d85e"
server_key_version: "encryptionkey@0"
I20260619 20:31:17.184778 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "d833ebb9b8b044b7a24a2ab1b09ba771"
format_stamp: "Formatted at 2026-06-19 20:31:17 on dist-test-slave-f9rk"
server_key: "9c73db9091e24c7474b406d5964582c9"
server_key_iv: "d2894c049fc97933e30ec3dee815d85e"
server_key_version: "encryptionkey@0"
I20260619 20:31:17.186070 10447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:17.186774 10462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:17.186952 10447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:17.187055 10447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "d833ebb9b8b044b7a24a2ab1b09ba771"
format_stamp: "Formatted at 2026-06-19 20:31:17 on dist-test-slave-f9rk"
server_key: "9c73db9091e24c7474b406d5964582c9"
server_key_iv: "d2894c049fc97933e30ec3dee815d85e"
server_key_version: "encryptionkey@0"
I20260619 20:31:17.187126 10447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:17.190420 10447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:17.192298 10447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:17.192451 10447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:17.196151 10447 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44971
I20260619 20:31:17.196182 10515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44971 every 8 connection(s)
I20260619 20:31:17.196516 10447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:17.197479 10516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:17.199733 10516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771: Bootstrap starting.
I20260619 20:31:17.200285 10516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:17.200572 10516 log.cc:824] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:17.201223 10516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771: No bootstrap required, opened a new log
I20260619 20:31:17.202754 10516 raft_consensus.cc:348] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d833ebb9b8b044b7a24a2ab1b09ba771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } }
I20260619 20:31:17.202883 10516 raft_consensus.cc:374] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:17.202932 10516 raft_consensus.cc:729] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d833ebb9b8b044b7a24a2ab1b09ba771, State: Initialized, Role: FOLLOWER
I20260619 20:31:17.203114 10516 consensus_queue.cc:260] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d833ebb9b8b044b7a24a2ab1b09ba771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } }
I20260619 20:31:17.203190 10516 raft_consensus.cc:388] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:17.203253 10516 raft_consensus.cc:482] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:17.203305 10516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:17.203866 10516 raft_consensus.cc:504] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d833ebb9b8b044b7a24a2ab1b09ba771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } }
I20260619 20:31:17.203990 10516 leader_election.cc:302] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d833ebb9b8b044b7a24a2ab1b09ba771; no voters:
I20260619 20:31:17.204159 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 10447
I20260619 20:31:17.204166 10516 leader_election.cc:288] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:17.204186 10519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:17.204283 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:17.204300 10519 raft_consensus.cc:686] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [term 1 LEADER]: Becoming Leader. State: Replica: d833ebb9b8b044b7a24a2ab1b09ba771, State: Running, Role: LEADER
I20260619 20:31:17.204377 10519 consensus_queue.cc:237] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d833ebb9b8b044b7a24a2ab1b09ba771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } }
I20260619 20:31:17.204422 10516 sys_catalog.cc:564] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:17.204476 22108 external_mini_cluster.cc:1428] Setting key b659f1babbc8665e5e9e2cffbc6fa8e3
I20260619 20:31:17.204655 10519 sys_catalog.cc:455] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d833ebb9b8b044b7a24a2ab1b09ba771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d833ebb9b8b044b7a24a2ab1b09ba771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } } }
I20260619 20:31:17.204732 10519 sys_catalog.cc:458] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:17.204846 10519 sys_catalog.cc:455] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d833ebb9b8b044b7a24a2ab1b09ba771. Latest consensus state: current_term: 1 leader_uuid: "d833ebb9b8b044b7a24a2ab1b09ba771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d833ebb9b8b044b7a24a2ab1b09ba771" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } } }
I20260619 20:31:17.204910 10519 sys_catalog.cc:458] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:17.205488 10524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:17.206017 10524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:17.206962 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:17.205124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48143 (local address 127.0.0.1:44971)
0619 20:31:17.205468 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:17.205479 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:17.205515 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:31:17.205632 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:17.205639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:17.205669 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:17.205825 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:17.206451 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:17.206454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:17.206455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:17.206458 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:17.206507 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:17.206523 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:17.206592 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:17.206790 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 20:31:17.206837 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":61,"threads_started":1}
I20260619 20:31:17.207959 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:17.208048 10524 catalog_manager.cc:1347] Generated new cluster ID: 330c54f5c100458e989035fa078ce5a1
I20260619 20:31:17.208097 10524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:17.210505 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:17.209383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37287 (local address 127.0.0.1:44971)
0619 20:31:17.209439 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:17.209441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:17.209549 (+ 108us) server_negotiation.cc:407] Connection header received
0619 20:31:17.209621 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:17.209622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:17.209627 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:17.209682 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:17.210263 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:17.210264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:17.210264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:17.210268 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:17.210282 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:17.210286 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:17.210338 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:17.210393 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 20:31:17.210407 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:31:17.218936 10524 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:17.219087 10524 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:17.222874 10524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d833ebb9b8b044b7a24a2ab1b09ba771: Generated new TSK 0
I20260619 20:31:17.223052 10524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:17.228813 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 10447
2026-06-19T20:31:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:17 dist-test-slave-f9rk krb5kdc[10547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:17 dist-test-slave-f9rk krb5kdc[10547](info): set up 2 sockets
Jun 19 20:31:17 dist-test-slave-f9rk krb5kdc[10547](info): commencing operation
krb5kdc: starting...
W20260619 20:31:18.276719 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:18 dist-test-slave-f9rk krb5kdc[10547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:18.390534 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:18.468931 10560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:18.469167 10560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:18.469204 10560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:18.470391 10560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:18.470471 10560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:18.470495 10560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:18.470516 10560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:18.470546 10560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:18.472003 10560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:18.472280 10560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:18.472539 10560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:18.477528 10566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:18.477521 10568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:18.477520 10565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:18.477998 10560 server_base.cc:1034] running on GCE node
I20260619 20:31:18.478216 10560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:18.478461 10560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:18.479598 10560 hybrid_clock.cc:648] HybridClock initialized: now 1781901078479461 us; error 151 us; skew 500 ppm
Jun 19 20:31:18 dist-test-slave-f9rk krb5kdc[10547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:18.482079 10560 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:18.482542 10560 webserver.cc:460] Webserver started at http://127.0.0.1:37613/ using document root <none> and password file <none>
I20260619 20:31:18.482780 10560 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:18.482834 10560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:18.482960 10560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:18.484097 10560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "28956338ad3b472394c2e7fdab730a92"
format_stamp: "Formatted at 2026-06-19 20:31:18 on dist-test-slave-f9rk"
server_key: "5c87adc6e6bf9ae8bc79fcab8239992a"
server_key_iv: "fb2741992a5526cf33c0fbc3ba2d41e1"
server_key_version: "encryptionkey@0"
I20260619 20:31:18.484452 10560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "28956338ad3b472394c2e7fdab730a92"
format_stamp: "Formatted at 2026-06-19 20:31:18 on dist-test-slave-f9rk"
server_key: "5c87adc6e6bf9ae8bc79fcab8239992a"
server_key_iv: "fb2741992a5526cf33c0fbc3ba2d41e1"
server_key_version: "encryptionkey@0"
I20260619 20:31:18.485757 10560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:31:18.486490 10575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:18.486685 10560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:18.486785 10560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "28956338ad3b472394c2e7fdab730a92"
format_stamp: "Formatted at 2026-06-19 20:31:18 on dist-test-slave-f9rk"
server_key: "5c87adc6e6bf9ae8bc79fcab8239992a"
server_key_iv: "fb2741992a5526cf33c0fbc3ba2d41e1"
server_key_version: "encryptionkey@0"
I20260619 20:31:18.486858 10560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:18.500965 10560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:18.502694 10560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:18.502848 10560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:18.506989 10560 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39537
I20260619 20:31:18.507020 10627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39537 every 8 connection(s)
I20260619 20:31:18.507354 10560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:18.508240 10628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:18.510471 10628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92: Bootstrap starting.
I20260619 20:31:18.511013 10628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:18.511253 10628 log.cc:824] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:18.511940 10628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92: No bootstrap required, opened a new log
I20260619 20:31:18.513429 10628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28956338ad3b472394c2e7fdab730a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260619 20:31:18.513558 10628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:18.513579 10628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28956338ad3b472394c2e7fdab730a92, State: Initialized, Role: FOLLOWER
I20260619 20:31:18.513675 10628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28956338ad3b472394c2e7fdab730a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260619 20:31:18.513724 10628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:18.513744 10628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:18.513769 10628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:18.514266 10628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28956338ad3b472394c2e7fdab730a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260619 20:31:18.514346 10628 leader_election.cc:302] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28956338ad3b472394c2e7fdab730a92; no voters:
I20260619 20:31:18.514489 10628 leader_election.cc:288] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:18.514544 10631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:18.514698 10631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [term 1 LEADER]: Becoming Leader. State: Replica: 28956338ad3b472394c2e7fdab730a92, State: Running, Role: LEADER
I20260619 20:31:18.514720 10628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:18.514847 10631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28956338ad3b472394c2e7fdab730a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260619 20:31:18.515225 10633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28956338ad3b472394c2e7fdab730a92. Latest consensus state: current_term: 1 leader_uuid: "28956338ad3b472394c2e7fdab730a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28956338ad3b472394c2e7fdab730a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } } }
I20260619 20:31:18.515339 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 10560
I20260619 20:31:18.515354 10632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28956338ad3b472394c2e7fdab730a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28956338ad3b472394c2e7fdab730a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } } }
I20260619 20:31:18.515370 10633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:18.515429 10632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:18.515462 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:18.515602 10635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:18.515690 22108 external_mini_cluster.cc:1428] Setting key 76ad87eccc95b0c29653d681a813b300
I20260619 20:31:18.516150 10635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:18.518014 10635 catalog_manager.cc:1347] Generated new cluster ID: aa868396456948d2893b758eed70d417
I20260619 20:31:18.518105 10635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:31:18 dist-test-slave-f9rk krb5kdc[10547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:31:18.527985 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:18.516400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:37241 (local address 127.0.0.1:39537)
0619 20:31:18.516558 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:18.516566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:18.516842 (+ 276us) server_negotiation.cc:407] Connection header received
0619 20:31:18.517054 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:18.517062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:18.517123 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:18.517252 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:18.518097 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:18.519511 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:18.520597 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:18.521236 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:18.521364 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:18.524211 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:18.524215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:18.524215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:18.524226 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:18.526179 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:18.526634 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:18.526637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:18.526637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:18.526736 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:18.526969 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:18.526970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:18.526970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:18.527247 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:18.527406 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:18.527533 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 20:31:18.527759 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260619 20:31:18.529138 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:18.530359 10635 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:18.531003 10635 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:18.533941 10635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28956338ad3b472394c2e7fdab730a92: Generated new TSK 0
I20260619 20:31:18.534090 10635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:18.536981 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:18.530512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:52543 (local address 127.0.0.1:39537)
0619 20:31:18.530560 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:18.530562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:18.530727 (+ 165us) server_negotiation.cc:407] Connection header received
0619 20:31:18.530768 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:18.530768 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:18.530778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:18.530818 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:18.531417 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:18.532023 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:18.532767 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:18.532983 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:18.533059 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:18.534396 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:18.534398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:18.534398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:18.534405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:18.535819 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:18.536168 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:18.536169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:18.536170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:18.536227 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:18.536497 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:18.536498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:18.536499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:18.536611 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:18.536718 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:18.536825 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 20:31:18.536882 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:31:18.538893 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 10560
W20260619 20:31:18.542497 10545 connection.cc:535] client connection to 127.0.0.1:39537 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39537): Connection reset by peer (error 104)
2026-06-19T20:31:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:18 dist-test-slave-f9rk krb5kdc[10659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:18 dist-test-slave-f9rk krb5kdc[10659](info): set up 2 sockets
Jun 19 20:31:18 dist-test-slave-f9rk krb5kdc[10659](info): commencing operation
krb5kdc: starting...
W20260619 20:31:19.576563 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:19 dist-test-slave-f9rk krb5kdc[10659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:19.693940 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:19.774658 10672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:19.774883 10672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:19.774916 10672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:19.776032 10672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:19.776109 10672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:19.776131 10672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:19.776150 10672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:19.776176 10672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:19.777640 10672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:19.777904 10672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:19.778141 10672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:19.782958 10680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:19.782958 10678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:19.782985 10672 server_base.cc:1034] running on GCE node
W20260619 20:31:19.782985 10677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:19.783427 10672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:19.783739 10672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:19.784879 10672 hybrid_clock.cc:648] HybridClock initialized: now 1781901079784874 us; error 33 us; skew 500 ppm
Jun 19 20:31:19 dist-test-slave-f9rk krb5kdc[10659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:19.787258 10672 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:19.787783 10672 webserver.cc:460] Webserver started at http://127.0.0.1:44819/ using document root <none> and password file <none>
I20260619 20:31:19.788019 10672 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:19.788081 10672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:19.788204 10672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:19.789247 10672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499"
format_stamp: "Formatted at 2026-06-19 20:31:19 on dist-test-slave-f9rk"
server_key: "7b07aedab0fedfa5d317ea190e295787"
server_key_iv: "4088af28d39b9210e0c667b64b4c072a"
server_key_version: "encryptionkey@0"
I20260619 20:31:19.789614 10672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499"
format_stamp: "Formatted at 2026-06-19 20:31:19 on dist-test-slave-f9rk"
server_key: "7b07aedab0fedfa5d317ea190e295787"
server_key_iv: "4088af28d39b9210e0c667b64b4c072a"
server_key_version: "encryptionkey@0"
I20260619 20:31:19.790858 10672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:31:19.791543 10687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:19.791729 10672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:19.791846 10672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499"
format_stamp: "Formatted at 2026-06-19 20:31:19 on dist-test-slave-f9rk"
server_key: "7b07aedab0fedfa5d317ea190e295787"
server_key_iv: "4088af28d39b9210e0c667b64b4c072a"
server_key_version: "encryptionkey@0"
I20260619 20:31:19.791944 10672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:19.802819 10672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:19.804545 10672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:19.804700 10672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:19.808670 10672 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42405
I20260619 20:31:19.808702 10739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42405 every 8 connection(s)
I20260619 20:31:19.809052 10672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:19.809899 10740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:19.812191 10740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499: Bootstrap starting.
I20260619 20:31:19.812740 10740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:19.813016 10740 log.cc:824] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:19.813668 10740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499: No bootstrap required, opened a new log
I20260619 20:31:19.815132 10740 raft_consensus.cc:348] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } }
I20260619 20:31:19.815259 10740 raft_consensus.cc:374] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:19.815280 10740 raft_consensus.cc:729] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da96d2f78a7f4a58a9e9aaa5f2ac7499, State: Initialized, Role: FOLLOWER
I20260619 20:31:19.815495 10740 consensus_queue.cc:260] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } }
I20260619 20:31:19.815558 10740 raft_consensus.cc:388] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:19.815639 10740 raft_consensus.cc:482] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:19.815694 10740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:19.816200 10740 raft_consensus.cc:504] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } }
I20260619 20:31:19.816306 10740 leader_election.cc:302] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da96d2f78a7f4a58a9e9aaa5f2ac7499; no voters:
I20260619 20:31:19.816515 10740 leader_election.cc:288] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:19.816550 10743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:19.816722 10743 raft_consensus.cc:686] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [term 1 LEADER]: Becoming Leader. State: Replica: da96d2f78a7f4a58a9e9aaa5f2ac7499, State: Running, Role: LEADER
I20260619 20:31:19.816763 10740 sys_catalog.cc:564] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:19.816836 10743 consensus_queue.cc:237] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } }
I20260619 20:31:19.817201 10744 sys_catalog.cc:455] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } } }
I20260619 20:31:19.817216 10745 sys_catalog.cc:455] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da96d2f78a7f4a58a9e9aaa5f2ac7499. Latest consensus state: current_term: 1 leader_uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da96d2f78a7f4a58a9e9aaa5f2ac7499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42405 } } }
I20260619 20:31:19.817382 10745 sys_catalog.cc:458] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:19.817368 10744 sys_catalog.cc:458] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:19.817764 10748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:19.818261 10748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:19.818693 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 10672
I20260619 20:31:19.818861 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:19.819036 22108 external_mini_cluster.cc:1428] Setting key 512d84f09ad4f58ff93dc03324037dad
I20260619 20:31:19.820207 10748 catalog_manager.cc:1347] Generated new cluster ID: 2f2cfc58cd984812af16c0db09e26c90
I20260619 20:31:19.820258 10748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:19.824194 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:19.819496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:35427 (local address 127.0.0.1:42405)
0619 20:31:19.819714 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:19.819723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:19.819765 (+ 42us) server_negotiation.cc:407] Connection header received
0619 20:31:19.819925 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:19.819932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:19.820001 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:19.820124 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:19.820425 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:19.821459 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:19.822232 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:19.822651 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:19.822770 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:19.823492 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:19.823495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:19.823496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:19.823749 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:19.823780 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:19.823792 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:19.823883 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:19.823964 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 20:31:19.823992 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260619 20:31:19.825286 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:19.828267 10748 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:19.828773 10748 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:19.831286 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:19.826588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:56009 (local address 127.0.0.1:42405)
0619 20:31:19.826631 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:19.826633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:19.827818 (+ 1185us) server_negotiation.cc:407] Connection header received
0619 20:31:19.828014 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:19.828015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:19.828022 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:19.828088 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:19.828222 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:19.828974 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:19.829946 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:19.830276 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:19.830350 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:19.831020 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:19.831021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:19.831022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:19.831026 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:19.831041 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:19.831045 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:19.831107 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:19.831199 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 20:31:19.831213 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:31:19.834893 10748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da96d2f78a7f4a58a9e9aaa5f2ac7499: Generated new TSK 0
I20260619 20:31:19.835054 10748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:19.839017 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 10672
W20260619 20:31:19.843009 10657 connection.cc:535] client connection to 127.0.0.1:42405 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42405): Connection reset by peer (error 104)
2026-06-19T20:31:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:19 dist-test-slave-f9rk krb5kdc[10771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:19 dist-test-slave-f9rk krb5kdc[10771](info): set up 2 sockets
Jun 19 20:31:19 dist-test-slave-f9rk krb5kdc[10771](info): commencing operation
krb5kdc: starting...
W20260619 20:31:20.872728 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:20 dist-test-slave-f9rk krb5kdc[10771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:20.986563 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:21.066802 10784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:21.067106 10784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:21.067147 10784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:21.069150 10784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:21.069203 10784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:21.069228 10784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:21.069252 10784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:21.069280 10784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:21.071550 10784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:21.071820 10784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:21.072065 10784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:21.076885 10789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:21.077018 10784 server_base.cc:1034] running on GCE node
W20260619 20:31:21.076872 10790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:21.077051 10792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:21.077334 10784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:21.077646 10784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:21.078807 10784 hybrid_clock.cc:648] HybridClock initialized: now 1781901081078792 us; error 27 us; skew 500 ppm
Jun 19 20:31:21 dist-test-slave-f9rk krb5kdc[10771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:21.085474 10784 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:21.085897 10784 webserver.cc:460] Webserver started at http://127.0.0.1:34985/ using document root <none> and password file <none>
I20260619 20:31:21.086151 10784 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:21.086220 10784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:21.086366 10784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:21.087397 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70"
format_stamp: "Formatted at 2026-06-19 20:31:21 on dist-test-slave-f9rk"
server_key: "9794fca2c2f58e3ce39613852f96f4c6"
server_key_iv: "688b12408bcd5be436420ac6d81a58e6"
server_key_version: "encryptionkey@0"
I20260619 20:31:21.087821 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70"
format_stamp: "Formatted at 2026-06-19 20:31:21 on dist-test-slave-f9rk"
server_key: "9794fca2c2f58e3ce39613852f96f4c6"
server_key_iv: "688b12408bcd5be436420ac6d81a58e6"
server_key_version: "encryptionkey@0"
I20260619 20:31:21.089128 10784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:31:21.089808 10799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:21.089994 10784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:21.090093 10784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70"
format_stamp: "Formatted at 2026-06-19 20:31:21 on dist-test-slave-f9rk"
server_key: "9794fca2c2f58e3ce39613852f96f4c6"
server_key_iv: "688b12408bcd5be436420ac6d81a58e6"
server_key_version: "encryptionkey@0"
I20260619 20:31:21.090178 10784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:21.093582 10784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:21.095340 10784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:21.095494 10784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:21.099155 10784 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36215
I20260619 20:31:21.099205 10851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36215 every 8 connection(s)
I20260619 20:31:21.099539 10784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:21.100557 10852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:21.101029 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 10784
I20260619 20:31:21.101159 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:21.101356 22108 external_mini_cluster.cc:1428] Setting key bdbed688e8dfa416c9bc39af05bcdeec
I20260619 20:31:21.103516 10852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70: Bootstrap starting.
I20260619 20:31:21.103933 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:21.101877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:45507 (local address 127.0.0.1:36215)
0619 20:31:21.102080 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:21.102089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:21.102124 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:31:21.102678 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:21.102686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:21.102722 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:21.102873 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:21.103465 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:21.103468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:21.103468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:21.103472 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:21.103511 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:21.103525 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:21.103601 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:21.103732 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 20:31:21.103778 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260619 20:31:21.104271 10852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:21.104579 10852 log.cc:824] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:21.104889 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:21.105335 10852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70: No bootstrap required, opened a new log
I20260619 20:31:21.107394 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:21.106453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:59665 (local address 127.0.0.1:36215)
0619 20:31:21.106489 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:21.106491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:21.106585 (+ 94us) server_negotiation.cc:407] Connection header received
0619 20:31:21.106636 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:21.106636 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:21.106640 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:21.106684 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:21.107193 (+ 509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:21.107194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:21.107194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:21.107198 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:21.107208 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:21.107212 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:21.107254 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:21.107313 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 20:31:21.107322 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:31:21.107982 10852 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } }
I20260619 20:31:21.108122 10852 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:21.108144 10852 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbcb2c805bef4ce792ec1c9f11ce6b70, State: Initialized, Role: FOLLOWER
I20260619 20:31:21.108218 10852 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } }
I20260619 20:31:21.108291 10852 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:21.108312 10852 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:21.108341 10852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:21.108803 10852 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } }
I20260619 20:31:21.108892 10852 leader_election.cc:302] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbcb2c805bef4ce792ec1c9f11ce6b70; no voters:
I20260619 20:31:21.109004 10852 leader_election.cc:288] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:21.109078 10862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:21.109244 10852 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:21.109262 10862 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [term 1 LEADER]: Becoming Leader. State: Replica: dbcb2c805bef4ce792ec1c9f11ce6b70, State: Running, Role: LEADER
I20260619 20:31:21.109344 10862 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } }
I20260619 20:31:21.109704 10864 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbcb2c805bef4ce792ec1c9f11ce6b70. Latest consensus state: current_term: 1 leader_uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } } }
I20260619 20:31:21.109820 10864 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:21.109704 10863 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcb2c805bef4ce792ec1c9f11ce6b70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36215 } } }
I20260619 20:31:21.109977 10863 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:21.110945 10870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:21.111424 10870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:21.112962 10870 catalog_manager.cc:1347] Generated new cluster ID: 70b2067fe9ec444baf8d05bf9832ed63
I20260619 20:31:21.113013 10870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:21.123790 10870 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:21.124020 10870 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:21.137997 10870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbcb2c805bef4ce792ec1c9f11ce6b70: Generated new TSK 0
I20260619 20:31:21.138236 10870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:21.140959 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 10784
2026-06-19T20:31:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:21 dist-test-slave-f9rk krb5kdc[10883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:21 dist-test-slave-f9rk krb5kdc[10883](info): set up 2 sockets
Jun 19 20:31:21 dist-test-slave-f9rk krb5kdc[10883](info): commencing operation
krb5kdc: starting...
W20260619 20:31:22.184855 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:22 dist-test-slave-f9rk krb5kdc[10883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:22.301751 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:22.382403 10896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:22.382623 10896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:22.382647 10896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:22.383905 10896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:22.383947 10896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:22.383962 10896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:22.383975 10896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:22.384018 10896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:22.385533 10896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:22.385802 10896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:22.386080 10896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:22.390936 10901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:22.390941 10904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:22.391074 10896 server_base.cc:1034] running on GCE node
W20260619 20:31:22.390960 10902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:22.391407 10896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:22.391712 10896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:22.392855 10896 hybrid_clock.cc:648] HybridClock initialized: now 1781901082392837 us; error 44 us; skew 500 ppm
Jun 19 20:31:22 dist-test-slave-f9rk krb5kdc[10883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:22.395263 10896 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:22.395819 10896 webserver.cc:460] Webserver started at http://127.0.0.1:42487/ using document root <none> and password file <none>
I20260619 20:31:22.396052 10896 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:22.396132 10896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:22.396251 10896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:22.397280 10896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "dc951511d6394c44b44a7e63d4eb2571"
format_stamp: "Formatted at 2026-06-19 20:31:22 on dist-test-slave-f9rk"
server_key: "d8d0cd9ef2bcae260ef7c7216eb9227c"
server_key_iv: "49e4fc2cc6f674d7959ba546b6fbc439"
server_key_version: "encryptionkey@0"
I20260619 20:31:22.397599 10896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "dc951511d6394c44b44a7e63d4eb2571"
format_stamp: "Formatted at 2026-06-19 20:31:22 on dist-test-slave-f9rk"
server_key: "d8d0cd9ef2bcae260ef7c7216eb9227c"
server_key_iv: "49e4fc2cc6f674d7959ba546b6fbc439"
server_key_version: "encryptionkey@0"
I20260619 20:31:22.398852 10896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:31:22.399538 10911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:22.399724 10896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:22.399823 10896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "dc951511d6394c44b44a7e63d4eb2571"
format_stamp: "Formatted at 2026-06-19 20:31:22 on dist-test-slave-f9rk"
server_key: "d8d0cd9ef2bcae260ef7c7216eb9227c"
server_key_iv: "49e4fc2cc6f674d7959ba546b6fbc439"
server_key_version: "encryptionkey@0"
I20260619 20:31:22.399910 10896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:22.412176 10896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:22.414563 10896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:22.414734 10896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:22.418967 10896 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42797
I20260619 20:31:22.419067 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42797 every 8 connection(s)
I20260619 20:31:22.419401 10896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:22.420305 10964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:22.422643 10964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571: Bootstrap starting.
I20260619 20:31:22.423226 10964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:22.423516 10964 log.cc:824] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:22.424203 10964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571: No bootstrap required, opened a new log
I20260619 20:31:22.426136 10964 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc951511d6394c44b44a7e63d4eb2571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260619 20:31:22.426301 10964 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:22.426419 10964 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc951511d6394c44b44a7e63d4eb2571, State: Initialized, Role: FOLLOWER
I20260619 20:31:22.426611 10964 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc951511d6394c44b44a7e63d4eb2571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260619 20:31:22.426682 10964 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:22.426705 10964 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:22.426726 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 10896
I20260619 20:31:22.426733 10964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:22.426874 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:22.427081 22108 external_mini_cluster.cc:1428] Setting key f2fae7b4d896840c24dded0b44930856
I20260619 20:31:22.427467 10964 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc951511d6394c44b44a7e63d4eb2571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260619 20:31:22.427593 10964 leader_election.cc:302] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc951511d6394c44b44a7e63d4eb2571; no voters:
I20260619 20:31:22.427799 10964 leader_election.cc:288] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:22.427953 10968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:22.428066 10968 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [term 1 LEADER]: Becoming Leader. State: Replica: dc951511d6394c44b44a7e63d4eb2571, State: Running, Role: LEADER
I20260619 20:31:22.428086 10964 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:22.428146 10968 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc951511d6394c44b44a7e63d4eb2571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } }
I20260619 20:31:22.428468 10968 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc951511d6394c44b44a7e63d4eb2571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc951511d6394c44b44a7e63d4eb2571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } } }
I20260619 20:31:22.428560 10968 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:22.428681 10968 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc951511d6394c44b44a7e63d4eb2571. Latest consensus state: current_term: 1 leader_uuid: "dc951511d6394c44b44a7e63d4eb2571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc951511d6394c44b44a7e63d4eb2571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42797 } } }
I20260619 20:31:22.428745 10968 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571 [sys.catalog]: This master's current role is: LEADER
W20260619 20:31:22.429935 10985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:31:22.429996 10985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:31:22.430048 10976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:22.430224 10976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:22.432161 10976 catalog_manager.cc:1347] Generated new cluster ID: a86745f5e9cc4c0ca93eeb2f39076f42
I20260619 20:31:22.432228 10976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:31:22 dist-test-slave-f9rk krb5kdc[10883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 20:31:22.440009 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:22.427667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:39139 (local address 127.0.0.1:42797)
0619 20:31:22.428068 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:22.428078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:22.428120 (+ 42us) server_negotiation.cc:407] Connection header received
0619 20:31:22.428271 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:22.428279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:22.428348 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:22.429398 (+ 1050us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:22.429516 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:22.430981 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:22.432105 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:22.432701 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:22.432831 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:22.435938 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:22.435943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:22.435946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:22.435960 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:22.438102 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:22.438594 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:22.438597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:22.438598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:22.438740 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:22.438986 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:22.438987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:22.438987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:22.439332 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:22.439520 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:22.439666 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 20:31:22.439778 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":123,"threads_started":1}
I20260619 20:31:22.441198 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:22.444484 10976 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:22.445185 10976 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:22.450258 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:22.442728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:33013 (local address 127.0.0.1:42797)
0619 20:31:22.442764 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:22.442765 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:22.442894 (+ 129us) server_negotiation.cc:407] Connection header received
0619 20:31:22.442956 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:22.442957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:22.442967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:22.443028 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:22.443769 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:22.444343 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:22.445388 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:22.445736 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:22.445808 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:22.447345 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:22.447346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:22.447347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:22.447353 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:22.448976 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:22.449412 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:22.449413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:22.449414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:22.449471 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:22.449715 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:22.449716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:22.449716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:22.449841 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:22.449961 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:22.450097 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 20:31:22.450163 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:31:22.451169 10976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc951511d6394c44b44a7e63d4eb2571: Generated new TSK 0
I20260619 20:31:22.451323 10976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:22.457247 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 10896
W20260619 20:31:22.461416 10881 connection.cc:535] client connection to 127.0.0.1:42797 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42797): Connection reset by peer (error 104)
2026-06-19T20:31:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:22 dist-test-slave-f9rk krb5kdc[10995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:22 dist-test-slave-f9rk krb5kdc[10995](info): set up 2 sockets
Jun 19 20:31:22 dist-test-slave-f9rk krb5kdc[10995](info): commencing operation
krb5kdc: starting...
W20260619 20:31:23.492817 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:23 dist-test-slave-f9rk krb5kdc[10995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:23.606499 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:23.686088 11008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:23.686280 11008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:23.686302 11008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:23.687464 11008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:23.687505 11008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:23.687520 11008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:23.687532 11008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:23.687585 11008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:23.689056 11008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:23.689307 11008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:23.689522 11008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:23.694325 11014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:23.694336 11016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:23.694344 11013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:23.694456 11008 server_base.cc:1034] running on GCE node
I20260619 20:31:23.694821 11008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:23.695148 11008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:23.696307 11008 hybrid_clock.cc:648] HybridClock initialized: now 1781901083696288 us; error 35 us; skew 500 ppm
Jun 19 20:31:23 dist-test-slave-f9rk krb5kdc[10995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:23.698594 11008 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:23.699021 11008 webserver.cc:460] Webserver started at http://127.0.0.1:45669/ using document root <none> and password file <none>
I20260619 20:31:23.699260 11008 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:23.699328 11008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:23.699484 11008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:23.700616 11008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "5bc6e58b5898425ab53585133d89c23f"
format_stamp: "Formatted at 2026-06-19 20:31:23 on dist-test-slave-f9rk"
server_key: "fa3b852843969a4420a2cdf8942cfe32"
server_key_iv: "c79ed2d9db3f9b8f8658f7a4e276d80e"
server_key_version: "encryptionkey@0"
I20260619 20:31:23.700985 11008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "5bc6e58b5898425ab53585133d89c23f"
format_stamp: "Formatted at 2026-06-19 20:31:23 on dist-test-slave-f9rk"
server_key: "fa3b852843969a4420a2cdf8942cfe32"
server_key_iv: "c79ed2d9db3f9b8f8658f7a4e276d80e"
server_key_version: "encryptionkey@0"
I20260619 20:31:23.702275 11008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:23.703025 11023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:23.703240 11008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 20:31:23.703339 11008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "5bc6e58b5898425ab53585133d89c23f"
format_stamp: "Formatted at 2026-06-19 20:31:23 on dist-test-slave-f9rk"
server_key: "fa3b852843969a4420a2cdf8942cfe32"
server_key_iv: "c79ed2d9db3f9b8f8658f7a4e276d80e"
server_key_version: "encryptionkey@0"
I20260619 20:31:23.703428 11008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:23.739326 11008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:23.741302 11008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:23.741472 11008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:23.745347 11008 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41113
I20260619 20:31:23.745378 11075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41113 every 8 connection(s)
I20260619 20:31:23.745741 11008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:23.746647 11076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:23.748881 11076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f: Bootstrap starting.
I20260619 20:31:23.749441 11076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:23.749720 11076 log.cc:824] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:23.750403 11076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f: No bootstrap required, opened a new log
I20260619 20:31:23.751582 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 11008
I20260619 20:31:23.751745 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:23.751966 22108 external_mini_cluster.cc:1428] Setting key d011af0269bcb06e0a88e7d2be06d418
I20260619 20:31:23.752394 11076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc6e58b5898425ab53585133d89c23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41113 } }
I20260619 20:31:23.752579 11076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:23.752621 11076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc6e58b5898425ab53585133d89c23f, State: Initialized, Role: FOLLOWER
I20260619 20:31:23.752756 11076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc6e58b5898425ab53585133d89c23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41113 } }
I20260619 20:31:23.752851 11076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:23.752895 11076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:23.752942 11076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:23.753786 11076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc6e58b5898425ab53585133d89c23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41113 } }
I20260619 20:31:23.753902 11076 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bc6e58b5898425ab53585133d89c23f; no voters:
I20260619 20:31:23.754077 11076 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:23.754094 11081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:23.754211 11081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [term 1 LEADER]: Becoming Leader. State: Replica: 5bc6e58b5898425ab53585133d89c23f, State: Running, Role: LEADER
I20260619 20:31:23.754290 11081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc6e58b5898425ab53585133d89c23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41113 } }
I20260619 20:31:23.754326 11076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:23.754601 11081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc6e58b5898425ab53585133d89c23f. Latest consensus state: current_term: 1 leader_uuid: "5bc6e58b5898425ab53585133d89c23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc6e58b5898425ab53585133d89c23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41113 } } }
I20260619 20:31:23.754676 11081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:23.754613 11082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bc6e58b5898425ab53585133d89c23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc6e58b5898425ab53585133d89c23f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41113 } } }
I20260619 20:31:23.754822 11082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:23.754889 11086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:23.755416 11086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:23.757469 11086 catalog_manager.cc:1347] Generated new cluster ID: 3619e8093c5b4c33a19e17a1418978b2
I20260619 20:31:23.757527 11086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:23.758008 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:23.752509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:39963 (local address 127.0.0.1:41113)
0619 20:31:23.752722 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:23.752731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:23.752762 (+ 31us) server_negotiation.cc:407] Connection header received
0619 20:31:23.752894 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:23.752901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:23.752970 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:23.753101 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:23.753307 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:23.754377 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:23.755416 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:23.756086 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:23.756188 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:23.757112 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:23.757116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:23.757116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:23.757319 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:23.757357 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:23.757371 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:23.757456 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:23.757752 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 20:31:23.757784 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260619 20:31:23.758971 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:23.763931 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:23.760371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:35693 (local address 127.0.0.1:41113)
0619 20:31:23.760410 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:23.760423 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:23.760578 (+ 155us) server_negotiation.cc:407] Connection header received
0619 20:31:23.760621 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:23.760622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:23.760633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:23.760698 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:23.760795 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:23.761510 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:23.762475 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:23.762858 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:23.762937 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:23.763668 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:23.763670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:23.763670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:23.763674 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:23.763689 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:23.763694 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:23.763759 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:23.763832 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 20:31:23.763844 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:31:23.767040 11086 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:23.767541 11086 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:23.773352 11086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bc6e58b5898425ab53585133d89c23f: Generated new TSK 0
I20260619 20:31:23.773514 11086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:23.776388 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 11008
W20260619 20:31:23.780550 10993 connection.cc:535] client connection to 127.0.0.1:41113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41113): Connection reset by peer (error 104)
2026-06-19T20:31:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:23 dist-test-slave-f9rk krb5kdc[11107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:23 dist-test-slave-f9rk krb5kdc[11107](info): set up 2 sockets
Jun 19 20:31:23 dist-test-slave-f9rk krb5kdc[11107](info): commencing operation
krb5kdc: starting...
W20260619 20:31:24.804973 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:24 dist-test-slave-f9rk krb5kdc[11107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 20:31:24.921861 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:25.003783 11120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:25.003994 11120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:25.004019 11120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:25.005220 11120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:25.005271 11120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:25.005288 11120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:25.005345 11120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:25.005367 11120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:25.006963 11120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:25.007238 11120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:25.007472 11120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 20:31:25.012400 11120 server_base.cc:1034] running on GCE node
W20260619 20:31:25.012347 11128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:25.012372 11126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:25.012368 11125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:25.012887 11120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:25.013175 11120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:25.014365 11120 hybrid_clock.cc:648] HybridClock initialized: now 1781901085014362 us; error 36 us; skew 500 ppm
Jun 19 20:31:25 dist-test-slave-f9rk krb5kdc[11107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:25.016937 11120 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 20:31:25.017446 11120 webserver.cc:460] Webserver started at http://127.0.0.1:44113/ using document root <none> and password file <none>
I20260619 20:31:25.017643 11120 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:25.017684 11120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:25.017769 11120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:25.018812 11120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "0ca381160b1e47359671715143bedef1"
format_stamp: "Formatted at 2026-06-19 20:31:25 on dist-test-slave-f9rk"
server_key: "4bf64190f8512d9e8e3bc2f3bdedff0b"
server_key_iv: "29320ecf5a55f5706ed3e73b068bf23d"
server_key_version: "encryptionkey@0"
I20260619 20:31:25.019117 11120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "0ca381160b1e47359671715143bedef1"
format_stamp: "Formatted at 2026-06-19 20:31:25 on dist-test-slave-f9rk"
server_key: "4bf64190f8512d9e8e3bc2f3bdedff0b"
server_key_iv: "29320ecf5a55f5706ed3e73b068bf23d"
server_key_version: "encryptionkey@0"
I20260619 20:31:25.020797 11120 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 20:31:25.021694 11135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:25.021893 11120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:31:25.021991 11120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "0ca381160b1e47359671715143bedef1"
format_stamp: "Formatted at 2026-06-19 20:31:25 on dist-test-slave-f9rk"
server_key: "4bf64190f8512d9e8e3bc2f3bdedff0b"
server_key_iv: "29320ecf5a55f5706ed3e73b068bf23d"
server_key_version: "encryptionkey@0"
I20260619 20:31:25.022049 11120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:25.025368 11120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:25.027760 11120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:25.027872 11120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:25.031481 11120 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42269
I20260619 20:31:25.031567 11187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42269 every 8 connection(s)
I20260619 20:31:25.031898 11120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:25.032925 11188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:25.035140 11188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1: Bootstrap starting.
I20260619 20:31:25.035754 11188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:25.036003 11188 log.cc:824] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:25.036695 11188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1: No bootstrap required, opened a new log
I20260619 20:31:25.036718 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 11120
I20260619 20:31:25.036827 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:25.037040 22108 external_mini_cluster.cc:1428] Setting key 61dc6bbad27b07b4a411e8d997c7d521
I20260619 20:31:25.038704 11188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca381160b1e47359671715143bedef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } }
I20260619 20:31:25.038893 11188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:25.038939 11188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca381160b1e47359671715143bedef1, State: Initialized, Role: FOLLOWER
I20260619 20:31:25.039125 11188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca381160b1e47359671715143bedef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } }
I20260619 20:31:25.039240 11188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:25.039314 11188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:25.039383 11188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:25.040042 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:25.037560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:38471 (local address 127.0.0.1:42269)
0619 20:31:25.037753 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:25.037761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:25.037787 (+ 26us) server_negotiation.cc:407] Connection header received
0619 20:31:25.038685 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:25.038691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:25.038717 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:25.038831 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:25.039552 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:25.039556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:25.039556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:25.039561 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:25.039655 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:25.039672 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:25.039736 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:25.039810 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 20:31:25.039854 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260619 20:31:25.040441 11188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca381160b1e47359671715143bedef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } }
I20260619 20:31:25.040558 11188 leader_election.cc:302] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ca381160b1e47359671715143bedef1; no voters:
I20260619 20:31:25.040714 11188 leader_election.cc:288] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:25.040733 11193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:25.040899 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:25.041002 11188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:25.042339 11193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [term 1 LEADER]: Becoming Leader. State: Replica: 0ca381160b1e47359671715143bedef1, State: Running, Role: LEADER
I20260619 20:31:25.042485 11193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca381160b1e47359671715143bedef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } }
I20260619 20:31:25.042977 11194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ca381160b1e47359671715143bedef1. Latest consensus state: current_term: 1 leader_uuid: "0ca381160b1e47359671715143bedef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca381160b1e47359671715143bedef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } } }
I20260619 20:31:25.043154 11194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:25.043066 11195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ca381160b1e47359671715143bedef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ca381160b1e47359671715143bedef1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } } }
W20260619 20:31:25.043407 11212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 20:31:25.043497 11195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1 [sys.catalog]: This master's current role is: LEADER
W20260619 20:31:25.043520 11212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:31:25.043599 11214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:25.043841 11214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:25.044498 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:25.042488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.126:49739 (local address 127.0.0.1:42269)
0619 20:31:25.042536 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:25.042538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:25.042742 (+ 204us) server_negotiation.cc:407] Connection header received
0619 20:31:25.042827 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:25.042828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:25.042833 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:25.042869 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:25.044266 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:25.044267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:25.044267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:25.044271 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:25.044287 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:25.044291 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:25.044356 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:25.044424 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 20:31:25.044437 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 20:31:25.045521 11214 catalog_manager.cc:1347] Generated new cluster ID: 0ac270a63f9945549d8213aaec907be0
I20260619 20:31:25.045570 11214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:25.054841 11214 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:25.055028 11214 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:25.059504 11214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ca381160b1e47359671715143bedef1: Generated new TSK 0
I20260619 20:31:25.059721 11214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:25.062951 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 11120
2026-06-19T20:31:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:25 dist-test-slave-f9rk krb5kdc[11219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:25 dist-test-slave-f9rk krb5kdc[11219](info): set up 2 sockets
Jun 19 20:31:25 dist-test-slave-f9rk krb5kdc[11219](info): commencing operation
krb5kdc: starting...
W20260619 20:31:26.096926 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:26 dist-test-slave-f9rk krb5kdc[11219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:26.211570 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:38623
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:33647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:26.293962 11232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:26.294195 11232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:26.294232 11232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:26.295396 11232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:26.295475 11232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:26.295502 11232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:26.295533 11232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:26.295563 11232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:26.297045 11232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:33647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:38623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:26.297318 11232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:26.297546 11232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:26.302548 11238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:26.302541 11237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:26.302580 11232 server_base.cc:1034] running on GCE node
W20260619 20:31:26.302541 11240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:26.303030 11232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:26.303339 11232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:26.304558 11232 hybrid_clock.cc:648] HybridClock initialized: now 1781901086304537 us; error 34 us; skew 500 ppm
Jun 19 20:31:26 dist-test-slave-f9rk krb5kdc[11219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901086, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:26.311491 11232 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:26.312004 11232 webserver.cc:460] Webserver started at http://127.21.151.62:42697/ using document root <none> and password file <none>
I20260619 20:31:26.312238 11232 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:26.312306 11232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:26.312460 11232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:26.313549 11232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "5618c204b9af423cab49a9c0b59fd322"
format_stamp: "Formatted at 2026-06-19 20:31:26 on dist-test-slave-f9rk"
server_key: "24bc8b6e1a595ffe73535ffd1b99c391"
server_key_iv: "4c827b715df6f98c9dbab53dcda3ef90"
server_key_version: "encryptionkey@0"
I20260619 20:31:26.313916 11232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "5618c204b9af423cab49a9c0b59fd322"
format_stamp: "Formatted at 2026-06-19 20:31:26 on dist-test-slave-f9rk"
server_key: "24bc8b6e1a595ffe73535ffd1b99c391"
server_key_iv: "4c827b715df6f98c9dbab53dcda3ef90"
server_key_version: "encryptionkey@0"
I20260619 20:31:26.315251 11232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:31:26.316092 11247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:26.316269 11232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:31:26.316370 11232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "5618c204b9af423cab49a9c0b59fd322"
format_stamp: "Formatted at 2026-06-19 20:31:26 on dist-test-slave-f9rk"
server_key: "24bc8b6e1a595ffe73535ffd1b99c391"
server_key_iv: "4c827b715df6f98c9dbab53dcda3ef90"
server_key_version: "encryptionkey@0"
I20260619 20:31:26.316443 11232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:26.333974 11232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:26.335804 11232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:26.335958 11232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:26.339531 11232 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:38623
I20260619 20:31:26.339601 11299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:38623 every 8 connection(s)
I20260619 20:31:26.339939 11232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:26.340797 11300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:26.342954 11300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322: Bootstrap starting.
I20260619 20:31:26.343469 11300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:26.343778 11300 log.cc:824] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:26.344410 11300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322: No bootstrap required, opened a new log
I20260619 20:31:26.345840 11300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5618c204b9af423cab49a9c0b59fd322" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38623 } }
I20260619 20:31:26.345978 11300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:26.346001 11300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5618c204b9af423cab49a9c0b59fd322, State: Initialized, Role: FOLLOWER
I20260619 20:31:26.346130 11300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5618c204b9af423cab49a9c0b59fd322" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38623 } }
I20260619 20:31:26.346208 11300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:26.346231 11300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:26.346258 11300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:26.346468 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 11232
I20260619 20:31:26.346599 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:26.346793 22108 external_mini_cluster.cc:1428] Setting key 0e96a144307375d4597975d731b3e9bb
I20260619 20:31:26.347255 11300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5618c204b9af423cab49a9c0b59fd322" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38623 } }
I20260619 20:31:26.347397 11300 leader_election.cc:302] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5618c204b9af423cab49a9c0b59fd322; no voters:
I20260619 20:31:26.347570 11300 leader_election.cc:288] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:26.347592 11305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:26.347842 11300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:26.347841 11305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [term 1 LEADER]: Becoming Leader. State: Replica: 5618c204b9af423cab49a9c0b59fd322, State: Running, Role: LEADER
I20260619 20:31:26.348577 11305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5618c204b9af423cab49a9c0b59fd322" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38623 } }
I20260619 20:31:26.349218 11306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5618c204b9af423cab49a9c0b59fd322" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5618c204b9af423cab49a9c0b59fd322" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38623 } } }
I20260619 20:31:26.349944 11306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:26.349368 11307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5618c204b9af423cab49a9c0b59fd322. Latest consensus state: current_term: 1 leader_uuid: "5618c204b9af423cab49a9c0b59fd322" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5618c204b9af423cab49a9c0b59fd322" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38623 } } }
I20260619 20:31:26.350126 11307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322 [sys.catalog]: This master's current role is: LEADER
W20260619 20:31:26.349685 11319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:31:26.350837 11319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:31:26.350948 11323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:26.351120 11323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:26.353564 11323 catalog_manager.cc:1347] Generated new cluster ID: abc56359f7e34cb1bf56f31f3253a3ec
I20260619 20:31:26.353618 11323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:31:26 dist-test-slave-f9rk krb5kdc[11219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:26.357563 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:26.347277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50353 (local address 127.21.151.62:38623)
0619 20:31:26.347482 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:26.347492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:26.347550 (+ 58us) server_negotiation.cc:407] Connection header received
0619 20:31:26.347699 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:26.347707 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:26.347764 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:26.347911 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:26.348639 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:26.350057 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:26.350808 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:26.351392 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:26.351498 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:26.354281 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:26.354285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:26.354285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:26.354296 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:26.356033 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:26.356457 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:26.356459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:26.356459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:26.356550 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:26.356779 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:26.356780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:26.356781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:26.357005 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:26.357182 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:26.357265 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 20:31:26.357390 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260619 20:31:26.358649 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:26.370124 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:26.363938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46253 (local address 127.21.151.62:38623)
0619 20:31:26.363977 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:26.363980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:26.364142 (+ 162us) server_negotiation.cc:407] Connection header received
0619 20:31:26.364215 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:26.364216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:26.364225 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:26.364282 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:26.364828 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:26.365296 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:26.366003 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:26.366221 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:26.366298 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:26.367460 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:26.367462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:26.367462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:26.367468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:26.368905 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:26.369297 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:26.369298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:26.369298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:26.369357 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:26.369597 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:26.369598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:26.369598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:26.369707 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:26.369817 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:26.369954 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 20:31:26.370016 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:31:26.372819 11323 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:26.373515 11323 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:26.380129 11323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5618c204b9af423cab49a9c0b59fd322: Generated new TSK 0
I20260619 20:31:26.380303 11323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:26.384379 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 11232
W20260619 20:31:26.388638 11217 connection.cc:535] client connection to 127.21.151.62:38623 recv error: Network error: failed to read from TLS socket (remote: 127.21.151.62:38623): Connection reset by peer (error 104)
2026-06-19T20:31:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:26 dist-test-slave-f9rk krb5kdc[11332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:26 dist-test-slave-f9rk krb5kdc[11332](info): set up 2 sockets
Jun 19 20:31:26 dist-test-slave-f9rk krb5kdc[11332](info): commencing operation
krb5kdc: starting...
W20260619 20:31:27.432808 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:27 dist-test-slave-f9rk krb5kdc[11332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:27.546532 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:42921
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:42921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:27.626320 11345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:27.626605 11345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:27.626644 11345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:27.627810 11345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:27.627887 11345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:27.627908 11345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:27.627928 11345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:27.627948 11345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:27.629431 11345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:42921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:42921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:27.629702 11345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:27.629927 11345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:27.634821 11351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:27.634810 11350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:27.634811 11353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:27.635023 11345 server_base.cc:1034] running on GCE node
I20260619 20:31:27.635288 11345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:27.635563 11345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:27.636760 11345 hybrid_clock.cc:648] HybridClock initialized: now 1781901087636744 us; error 29 us; skew 500 ppm
Jun 19 20:31:27 dist-test-slave-f9rk krb5kdc[11332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901087, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:27.639068 11345 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:27.639487 11345 webserver.cc:460] Webserver started at http://127.21.151.62:34629/ using document root <none> and password file <none>
I20260619 20:31:27.639762 11345 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:27.639832 11345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:27.639997 11345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:27.641130 11345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "c183236208c543418f4deeb9670e250a"
format_stamp: "Formatted at 2026-06-19 20:31:27 on dist-test-slave-f9rk"
server_key: "ca0f007f8c20dd73fdc9e5ca8f53120a"
server_key_iv: "80d8ad228395920477d2c221c8736d07"
server_key_version: "encryptionkey@0"
I20260619 20:31:27.641517 11345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "c183236208c543418f4deeb9670e250a"
format_stamp: "Formatted at 2026-06-19 20:31:27 on dist-test-slave-f9rk"
server_key: "ca0f007f8c20dd73fdc9e5ca8f53120a"
server_key_iv: "80d8ad228395920477d2c221c8736d07"
server_key_version: "encryptionkey@0"
I20260619 20:31:27.642930 11345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:31:27.643689 11360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:27.643863 11345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:27.643961 11345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "c183236208c543418f4deeb9670e250a"
format_stamp: "Formatted at 2026-06-19 20:31:27 on dist-test-slave-f9rk"
server_key: "ca0f007f8c20dd73fdc9e5ca8f53120a"
server_key_iv: "80d8ad228395920477d2c221c8736d07"
server_key_version: "encryptionkey@0"
I20260619 20:31:27.644049 11345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:27.653916 11345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:27.655787 11345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:27.655951 11345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:27.659741 11345 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:42921
I20260619 20:31:27.659799 11412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:42921 every 8 connection(s)
I20260619 20:31:27.660161 11345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:27.660974 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 11345
I20260619 20:31:27.661008 11413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:27.661100 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:27.661298 22108 external_mini_cluster.cc:1428] Setting key e0252a55a60af759d7e3cfe0a5793820
I20260619 20:31:27.664233 11413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a: Bootstrap starting.
I20260619 20:31:27.664916 11413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:27.665195 11413 log.cc:824] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:27.665892 11413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a: No bootstrap required, opened a new log
I20260619 20:31:27.667040 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:27.661933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36819 (local address 127.21.151.62:42921)
0619 20:31:27.662090 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:27.662098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:27.662122 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:31:27.662569 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:27.662575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:27.662621 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:27.662783 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:27.663003 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:27.663939 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:27.665045 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:27.665477 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:27.665601 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:27.666337 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:27.666342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:27.666343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:27.666544 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:27.666582 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:27.666596 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:27.666683 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:27.666792 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 20:31:27.666820 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260619 20:31:27.668170 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:27.668205 11413 raft_consensus.cc:348] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c183236208c543418f4deeb9670e250a" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42921 } }
I20260619 20:31:27.668363 11413 raft_consensus.cc:374] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:27.668390 11413 raft_consensus.cc:729] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c183236208c543418f4deeb9670e250a, State: Initialized, Role: FOLLOWER
I20260619 20:31:27.668507 11413 consensus_queue.cc:260] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c183236208c543418f4deeb9670e250a" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42921 } }
I20260619 20:31:27.668570 11413 raft_consensus.cc:388] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:27.668598 11413 raft_consensus.cc:482] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:27.668649 11413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:27.669381 11413 raft_consensus.cc:504] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c183236208c543418f4deeb9670e250a" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42921 } }
I20260619 20:31:27.669519 11413 leader_election.cc:302] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c183236208c543418f4deeb9670e250a; no voters:
I20260619 20:31:27.669682 11413 leader_election.cc:288] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:27.669716 11423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:27.669860 11423 raft_consensus.cc:686] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [term 1 LEADER]: Becoming Leader. State: Replica: c183236208c543418f4deeb9670e250a, State: Running, Role: LEADER
I20260619 20:31:27.669966 11413 sys_catalog.cc:564] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:27.669989 11423 consensus_queue.cc:237] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c183236208c543418f4deeb9670e250a" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42921 } }
I20260619 20:31:27.670369 11426 sys_catalog.cc:455] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c183236208c543418f4deeb9670e250a. Latest consensus state: current_term: 1 leader_uuid: "c183236208c543418f4deeb9670e250a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c183236208c543418f4deeb9670e250a" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42921 } } }
I20260619 20:31:27.670403 11425 sys_catalog.cc:455] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c183236208c543418f4deeb9670e250a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c183236208c543418f4deeb9670e250a" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 42921 } } }
I20260619 20:31:27.670451 11426 sys_catalog.cc:458] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:27.670547 11425 sys_catalog.cc:458] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:27.670737 11432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:27.671224 11432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:27.672899 11432 catalog_manager.cc:1347] Generated new cluster ID: def317ca5b644c038bd3eeb8a1184fc1
I20260619 20:31:27.672948 11432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:27.674806 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:27.669683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49089 (local address 127.21.151.62:42921)
0619 20:31:27.669723 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:27.669725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:27.671512 (+ 1787us) server_negotiation.cc:407] Connection header received
0619 20:31:27.671604 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:27.671605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:27.671634 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:27.671664 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:27.671904 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:27.672480 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:27.673483 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:27.673779 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:27.673843 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:27.674565 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:27.674566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:27.674566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:27.674570 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:27.674583 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:27.674587 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:27.674638 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:27.674723 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:31:27.674733 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:31:27.691689 11432 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:27.692229 11432 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:27.697886 11432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c183236208c543418f4deeb9670e250a: Generated new TSK 0
I20260619 20:31:27.698094 11432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:27.700519 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 11345
W20260619 20:31:27.704954 11330 connection.cc:535] client connection to 127.21.151.62:42921 recv error: Network error: failed to read from TLS socket (remote: 127.21.151.62:42921): Connection reset by peer (error 104)
2026-06-19T20:31:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:27 dist-test-slave-f9rk krb5kdc[11445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:27 dist-test-slave-f9rk krb5kdc[11445](info): set up 2 sockets
Jun 19 20:31:27 dist-test-slave-f9rk krb5kdc[11445](info): commencing operation
krb5kdc: starting...
W20260619 20:31:28.744817 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:28 dist-test-slave-f9rk krb5kdc[11445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:28.860306 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:46089
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:33925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:46089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:28.940882 11458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:28.941090 11458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:28.941113 11458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:28.942234 11458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:28.942269 11458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:28.942283 11458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:28.942296 11458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:28.942308 11458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:28.943742 11458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:33925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:46089
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:46089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:28.943943 11458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:28.944118 11458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:28.949225 11463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:28.949261 11464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:28.949265 11466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:28.949535 11458 server_base.cc:1034] running on GCE node
I20260619 20:31:28.949781 11458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:28.950093 11458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:28.951239 11458 hybrid_clock.cc:648] HybridClock initialized: now 1781901088951238 us; error 52 us; skew 500 ppm
Jun 19 20:31:28 dist-test-slave-f9rk krb5kdc[11445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901088, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:28.953801 11458 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:28.954282 11458 webserver.cc:460] Webserver started at http://127.21.151.62:42077/ using document root <none> and password file <none>
I20260619 20:31:28.954531 11458 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:28.954602 11458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:28.954758 11458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:28.955895 11458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "cdaa7bfad31d479ba47ac468fb59addc"
format_stamp: "Formatted at 2026-06-19 20:31:28 on dist-test-slave-f9rk"
server_key: "8df77659312d34b96fb337d3a6db13de"
server_key_iv: "eeda3e0251331525b9b218ea6c0fdc1d"
server_key_version: "encryptionkey@0"
I20260619 20:31:28.956279 11458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "cdaa7bfad31d479ba47ac468fb59addc"
format_stamp: "Formatted at 2026-06-19 20:31:28 on dist-test-slave-f9rk"
server_key: "8df77659312d34b96fb337d3a6db13de"
server_key_iv: "eeda3e0251331525b9b218ea6c0fdc1d"
server_key_version: "encryptionkey@0"
I20260619 20:31:28.957612 11458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 20:31:28.958341 11473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:28.958549 11458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:28.958637 11458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "cdaa7bfad31d479ba47ac468fb59addc"
format_stamp: "Formatted at 2026-06-19 20:31:28 on dist-test-slave-f9rk"
server_key: "8df77659312d34b96fb337d3a6db13de"
server_key_iv: "eeda3e0251331525b9b218ea6c0fdc1d"
server_key_version: "encryptionkey@0"
I20260619 20:31:28.958694 11458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:28.961942 11458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:28.963933 11458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:28.964040 11458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:28.968544 11458 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:46089
I20260619 20:31:28.968570 11525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:46089 every 8 connection(s)
I20260619 20:31:28.968919 11458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:28.969807 11526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:28.972114 11526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc: Bootstrap starting.
I20260619 20:31:28.972637 11526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:28.972877 11526 log.cc:824] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:28.973510 11526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc: No bootstrap required, opened a new log
I20260619 20:31:28.975186 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 11458
I20260619 20:31:28.975075 11526 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdaa7bfad31d479ba47ac468fb59addc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46089 } }
I20260619 20:31:28.975270 11526 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:28.975302 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:28.975329 11526 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaa7bfad31d479ba47ac468fb59addc, State: Initialized, Role: FOLLOWER
I20260619 20:31:28.975507 22108 external_mini_cluster.cc:1428] Setting key a7dd5c731b071e9345991df98cf139f4
I20260619 20:31:28.975505 11526 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdaa7bfad31d479ba47ac468fb59addc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46089 } }
I20260619 20:31:28.975579 11526 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:28.975612 11526 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:28.975749 11526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:28.977154 11526 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdaa7bfad31d479ba47ac468fb59addc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46089 } }
I20260619 20:31:28.977305 11526 leader_election.cc:302] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdaa7bfad31d479ba47ac468fb59addc; no voters:
I20260619 20:31:28.977474 11526 leader_election.cc:288] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:28.977506 11531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:28.977698 11531 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [term 1 LEADER]: Becoming Leader. State: Replica: cdaa7bfad31d479ba47ac468fb59addc, State: Running, Role: LEADER
I20260619 20:31:28.977753 11526 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:28.977851 11531 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdaa7bfad31d479ba47ac468fb59addc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46089 } }
I20260619 20:31:28.978108 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:28.976023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53105 (local address 127.21.151.62:46089)
0619 20:31:28.976387 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:28.976398 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:28.976438 (+ 40us) server_negotiation.cc:407] Connection header received
0619 20:31:28.976593 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:28.976600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:28.976632 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:28.976783 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:28.977442 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:28.977445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:28.977446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:28.977450 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:28.977498 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:28.977514 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:28.977835 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:28.977867 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 20:31:28.977913 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":75,"threads_started":1}
I20260619 20:31:28.978305 11532 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdaa7bfad31d479ba47ac468fb59addc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdaa7bfad31d479ba47ac468fb59addc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46089 } } }
I20260619 20:31:28.978423 11532 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:28.978212 11531 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdaa7bfad31d479ba47ac468fb59addc. Latest consensus state: current_term: 1 leader_uuid: "cdaa7bfad31d479ba47ac468fb59addc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdaa7bfad31d479ba47ac468fb59addc" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 46089 } } }
I20260619 20:31:28.978699 11531 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:28.978910 11540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:28.979207 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:28.979440 11540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:28.981225 11540 catalog_manager.cc:1347] Generated new cluster ID: 39df00f8fa9a4fad8d2ff87cd02a48a6
I20260619 20:31:28.981285 11540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:28.981990 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:28.980805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42109 (local address 127.21.151.62:46089)
0619 20:31:28.980840 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:28.980842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:28.980940 (+ 98us) server_negotiation.cc:407] Connection header received
0619 20:31:28.981011 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:28.981012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:28.981028 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:28.981084 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:28.981771 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:28.981772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:28.981772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:28.981777 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:28.981789 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:28.981793 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:28.981820 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:28.981913 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 20:31:28.981926 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:31:28.992784 11540 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:28.992954 11540 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:28.996557 11540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdaa7bfad31d479ba47ac468fb59addc: Generated new TSK 0
I20260619 20:31:28.996748 11540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:29.005254 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 11458
2026-06-19T20:31:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:29 dist-test-slave-f9rk krb5kdc[11557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:29 dist-test-slave-f9rk krb5kdc[11557](info): set up 2 sockets
Jun 19 20:31:29 dist-test-slave-f9rk krb5kdc[11557](info): commencing operation
krb5kdc: starting...
W20260619 20:31:30.064781 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:30 dist-test-slave-f9rk krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:30.180004 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:43823
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:43823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:30.260104 11570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:30.260315 11570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:30.260339 11570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:30.261588 11570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:30.261650 11570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:30.261664 11570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:30.261677 11570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:30.261751 11570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:30.263221 11570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:43823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:43823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:30.263507 11570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:30.263809 11570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:30.268714 11578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:30.268582 11575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:30.268544 11576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:30.268896 11570 server_base.cc:1034] running on GCE node
I20260619 20:31:30.269112 11570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:30.269402 11570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:30.270550 11570 hybrid_clock.cc:648] HybridClock initialized: now 1781901090270530 us; error 39 us; skew 500 ppm
Jun 19 20:31:30 dist-test-slave-f9rk krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901090, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:30.273165 11570 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:30.273574 11570 webserver.cc:460] Webserver started at http://127.21.151.62:39933/ using document root <none> and password file <none>
I20260619 20:31:30.273754 11570 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:30.273797 11570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:30.273882 11570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:30.274966 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "029071d52f9443e18f5bd89cb68451bd"
format_stamp: "Formatted at 2026-06-19 20:31:30 on dist-test-slave-f9rk"
server_key: "8bc2991649c71649d97e7c1d83ca48b3"
server_key_iv: "0df6f822cf385f69c96d23489b18d2b4"
server_key_version: "encryptionkey@0"
I20260619 20:31:30.275278 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "029071d52f9443e18f5bd89cb68451bd"
format_stamp: "Formatted at 2026-06-19 20:31:30 on dist-test-slave-f9rk"
server_key: "8bc2991649c71649d97e7c1d83ca48b3"
server_key_iv: "0df6f822cf385f69c96d23489b18d2b4"
server_key_version: "encryptionkey@0"
I20260619 20:31:30.276623 11570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:31:30.277347 11585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:30.277570 11570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 20:31:30.277653 11570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "029071d52f9443e18f5bd89cb68451bd"
format_stamp: "Formatted at 2026-06-19 20:31:30 on dist-test-slave-f9rk"
server_key: "8bc2991649c71649d97e7c1d83ca48b3"
server_key_iv: "0df6f822cf385f69c96d23489b18d2b4"
server_key_version: "encryptionkey@0"
I20260619 20:31:30.277711 11570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:30.291163 11570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:30.293031 11570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:30.293143 11570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:30.296936 11570 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:43823
I20260619 20:31:30.297004 11637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:43823 every 8 connection(s)
I20260619 20:31:30.297345 11570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:30.298439 11638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:30.301579 11638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd: Bootstrap starting.
I20260619 20:31:30.302165 11638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:30.302452 11638 log.cc:824] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:30.303140 11638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd: No bootstrap required, opened a new log
I20260619 20:31:30.304728 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 11570
I20260619 20:31:30.304850 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:30.304792 11638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029071d52f9443e18f5bd89cb68451bd" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43823 } }
I20260619 20:31:30.305009 11638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:30.305052 22108 external_mini_cluster.cc:1428] Setting key a1e8b33c63ed3c63f3545637a9e06299
I20260619 20:31:30.305052 11638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 029071d52f9443e18f5bd89cb68451bd, State: Initialized, Role: FOLLOWER
I20260619 20:31:30.305215 11638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029071d52f9443e18f5bd89cb68451bd" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43823 } }
I20260619 20:31:30.305308 11638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:30.305331 11638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:30.305398 11638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:30.306308 11638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029071d52f9443e18f5bd89cb68451bd" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43823 } }
I20260619 20:31:30.306429 11638 leader_election.cc:302] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 029071d52f9443e18f5bd89cb68451bd; no voters:
I20260619 20:31:30.306643 11638 leader_election.cc:288] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:30.306663 11643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:30.306775 11643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [term 1 LEADER]: Becoming Leader. State: Replica: 029071d52f9443e18f5bd89cb68451bd, State: Running, Role: LEADER
I20260619 20:31:30.306895 11638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:30.306943 11643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029071d52f9443e18f5bd89cb68451bd" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43823 } }
I20260619 20:31:30.307391 11645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "029071d52f9443e18f5bd89cb68451bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029071d52f9443e18f5bd89cb68451bd" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43823 } } }
I20260619 20:31:30.307472 11645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:30.307729 11644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 029071d52f9443e18f5bd89cb68451bd. Latest consensus state: current_term: 1 leader_uuid: "029071d52f9443e18f5bd89cb68451bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029071d52f9443e18f5bd89cb68451bd" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 43823 } } }
I20260619 20:31:30.307809 11644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:30.307739 11651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:30.308393 11651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:30.310312 11651 catalog_manager.cc:1347] Generated new cluster ID: f93b76ddea3b4040b3f21313fa3e69b8
I20260619 20:31:30.310370 11651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 20:31:30 dist-test-slave-f9rk krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:30.316558 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:30.305552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33745 (local address 127.21.151.62:43823)
0619 20:31:30.305724 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:30.305734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:30.305770 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:31:30.305910 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:30.305916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:30.305958 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:30.306095 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:30.306873 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:30.308300 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:30.309341 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:30.309870 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:30.309989 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:30.312847 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:30.312852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:30.312852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:30.312864 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:30.314833 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:30.315228 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:30.315232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:30.315232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:30.315395 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:30.315609 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:30.315610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:30.315611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:30.315953 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:30.316123 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:30.316194 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 20:31:30.316348 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260619 20:31:30.317571 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:30.325899 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:30.319031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42807 (local address 127.21.151.62:43823)
0619 20:31:30.319065 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:30.319066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:30.319092 (+ 26us) server_negotiation.cc:407] Connection header received
0619 20:31:30.319148 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:30.319148 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:30.319158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:30.319219 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:30.319825 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:30.320592 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:30.321540 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:30.321860 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:30.321929 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:30.323226 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:30.323227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:30.323227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:30.323235 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:30.324796 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:30.325158 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:30.325160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:30.325160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:30.325217 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:30.325412 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:30.325413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:30.325413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:30.325537 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:30.325641 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:30.325733 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 20:31:30.325807 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:31:30.336479 11651 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:30.337195 11651 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:30.346040 11651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 029071d52f9443e18f5bd89cb68451bd: Generated new TSK 0
I20260619 20:31:30.346244 11651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:30.350100 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 11570
2026-06-19T20:31:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:30 dist-test-slave-f9rk krb5kdc[11669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:30 dist-test-slave-f9rk krb5kdc[11669](info): set up 2 sockets
Jun 19 20:31:30 dist-test-slave-f9rk krb5kdc[11669](info): commencing operation
krb5kdc: starting...
W20260619 20:31:31.400794 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:31 dist-test-slave-f9rk krb5kdc[11669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:31.518129 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:38721
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:38721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:31.598647 11682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:31.598872 11682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:31.598896 11682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:31.600042 11682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:31.600080 11682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:31.600095 11682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:31.600108 11682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:31.600121 11682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:31.601588 11682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:38721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:38721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:31.601785 11682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:31.601958 11682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:31.606760 11688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:31.606798 11682 server_base.cc:1034] running on GCE node
W20260619 20:31:31.606832 11687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:31.606770 11690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:31.607159 11682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:31.607470 11682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:31.608640 11682 hybrid_clock.cc:648] HybridClock initialized: now 1781901091608614 us; error 38 us; skew 500 ppm
Jun 19 20:31:31 dist-test-slave-f9rk krb5kdc[11669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901091, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:31.616285 11682 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:31.616740 11682 webserver.cc:460] Webserver started at http://127.21.151.62:33549/ using document root <none> and password file <none>
I20260619 20:31:31.616984 11682 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:31.617054 11682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:31.617211 11682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:31.618333 11682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "6bc2a04b3fab482d93e631e25bf33ca7"
format_stamp: "Formatted at 2026-06-19 20:31:31 on dist-test-slave-f9rk"
server_key: "858eef38c3e830601a413fe5660f2b13"
server_key_iv: "b70db29af897ab613c59239a95c8628b"
server_key_version: "encryptionkey@0"
I20260619 20:31:31.618701 11682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "6bc2a04b3fab482d93e631e25bf33ca7"
format_stamp: "Formatted at 2026-06-19 20:31:31 on dist-test-slave-f9rk"
server_key: "858eef38c3e830601a413fe5660f2b13"
server_key_iv: "b70db29af897ab613c59239a95c8628b"
server_key_version: "encryptionkey@0"
I20260619 20:31:31.620100 11682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:31:31.620837 11697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:31.621009 11682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:31.621115 11682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "6bc2a04b3fab482d93e631e25bf33ca7"
format_stamp: "Formatted at 2026-06-19 20:31:31 on dist-test-slave-f9rk"
server_key: "858eef38c3e830601a413fe5660f2b13"
server_key_iv: "b70db29af897ab613c59239a95c8628b"
server_key_version: "encryptionkey@0"
I20260619 20:31:31.621189 11682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:31.631541 11682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:31.633543 11682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:31.633715 11682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:31.637712 11682 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:38721
I20260619 20:31:31.637748 11749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:38721 every 8 connection(s)
I20260619 20:31:31.638124 11682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:31.639062 11750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:31.641404 11750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7: Bootstrap starting.
I20260619 20:31:31.641932 11750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:31.642156 11750 log.cc:824] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:31.642827 11750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7: No bootstrap required, opened a new log
I20260619 20:31:31.643050 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 11682
I20260619 20:31:31.643173 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:31.643358 22108 external_mini_cluster.cc:1428] Setting key afa4c512e9c21a4a306b15cf4c250139
I20260619 20:31:31.645130 11750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc2a04b3fab482d93e631e25bf33ca7" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38721 } }
I20260619 20:31:31.645277 11750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:31.645300 11750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc2a04b3fab482d93e631e25bf33ca7, State: Initialized, Role: FOLLOWER
I20260619 20:31:31.645434 11750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc2a04b3fab482d93e631e25bf33ca7" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38721 } }
I20260619 20:31:31.645507 11750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:31.645529 11750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:31.645601 11750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:31.646158 11750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc2a04b3fab482d93e631e25bf33ca7" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38721 } }
I20260619 20:31:31.646243 11750 leader_election.cc:302] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bc2a04b3fab482d93e631e25bf33ca7; no voters:
I20260619 20:31:31.646432 11750 leader_election.cc:288] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:31.646464 11755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:31.646626 11755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [term 1 LEADER]: Becoming Leader. State: Replica: 6bc2a04b3fab482d93e631e25bf33ca7, State: Running, Role: LEADER
I20260619 20:31:31.646687 11750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:31.646754 11755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc2a04b3fab482d93e631e25bf33ca7" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38721 } }
I20260619 20:31:31.647105 11755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bc2a04b3fab482d93e631e25bf33ca7. Latest consensus state: current_term: 1 leader_uuid: "6bc2a04b3fab482d93e631e25bf33ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc2a04b3fab482d93e631e25bf33ca7" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38721 } } }
I20260619 20:31:31.647119 11757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bc2a04b3fab482d93e631e25bf33ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc2a04b3fab482d93e631e25bf33ca7" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 38721 } } }
I20260619 20:31:31.647212 11755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:31.647228 11757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:31.647589 11762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:31.648149 11762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:31.650036 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:31.643909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59185 (local address 127.21.151.62:38721)
0619 20:31:31.644183 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:31.644192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:31.644228 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:31:31.644631 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:31.644638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:31.644685 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:31.644800 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:31.645060 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:31.646436 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:31.647582 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:31.648310 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:31.648491 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:31.649329 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:31.649332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:31.649333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:31.649533 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:31.649571 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:31.649586 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:31.649661 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:31.649746 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 20:31:31.649813 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20260619 20:31:31.651051 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:31.652763 11762 catalog_manager.cc:1347] Generated new cluster ID: f4e5d58eec9942699f609353c2cb2f2a
I20260619 20:31:31.652822 11762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:31.658762 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:31.655902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58749 (local address 127.21.151.62:38721)
0619 20:31:31.655942 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:31.655944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:31.656070 (+ 126us) server_negotiation.cc:407] Connection header received
0619 20:31:31.656145 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:31.656146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:31.656155 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:31.656200 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:31.656322 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:31.656788 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:31.657506 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:31.657719 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:31.657806 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:31.658505 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:31.658506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:31.658507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:31.658511 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:31.658526 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:31.658531 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:31.658584 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:31.658643 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 20:31:31.658665 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:31:31.668193 11762 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:31.668924 11762 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:31.674418 11762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bc2a04b3fab482d93e631e25bf33ca7: Generated new TSK 0
I20260619 20:31:31.674607 11762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:31.678747 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 11682
2026-06-19T20:31:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:31 dist-test-slave-f9rk krb5kdc[11781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:31 dist-test-slave-f9rk krb5kdc[11781](info): set up 2 sockets
Jun 19 20:31:31 dist-test-slave-f9rk krb5kdc[11781](info): commencing operation
krb5kdc: starting...
W20260619 20:31:32.725034 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:32 dist-test-slave-f9rk krb5kdc[11781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:32.838505 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:36507
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:36507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:32.918725 11794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:32.918977 11794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:32.919013 11794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:32.920202 11794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:32.920280 11794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:32.920305 11794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:32.920326 11794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:32.920352 11794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:32.921834 11794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:36507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:36507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:32.922132 11794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:32.922363 11794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:32.927078 11800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:32.927109 11802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:32.927193 11794 server_base.cc:1034] running on GCE node
W20260619 20:31:32.927119 11799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:32.927539 11794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:32.927834 11794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:32.928972 11794 hybrid_clock.cc:648] HybridClock initialized: now 1781901092928957 us; error 30 us; skew 500 ppm
Jun 19 20:31:32 dist-test-slave-f9rk krb5kdc[11781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901092, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:32.931291 11794 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:32.931722 11794 webserver.cc:460] Webserver started at http://127.21.151.62:43565/ using document root <none> and password file <none>
I20260619 20:31:32.931905 11794 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:32.931946 11794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:32.932034 11794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:32.933096 11794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "020ed78588d44814be7bd99423fc9e22"
format_stamp: "Formatted at 2026-06-19 20:31:32 on dist-test-slave-f9rk"
server_key: "6b13f9bf4169820b47e82773585f8324"
server_key_iv: "cc86c92feaa32cfacf2184b65086007c"
server_key_version: "encryptionkey@0"
I20260619 20:31:32.933400 11794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "020ed78588d44814be7bd99423fc9e22"
format_stamp: "Formatted at 2026-06-19 20:31:32 on dist-test-slave-f9rk"
server_key: "6b13f9bf4169820b47e82773585f8324"
server_key_iv: "cc86c92feaa32cfacf2184b65086007c"
server_key_version: "encryptionkey@0"
I20260619 20:31:32.934614 11794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 20:31:32.935284 11809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:32.935465 11794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 20:31:32.935549 11794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "020ed78588d44814be7bd99423fc9e22"
format_stamp: "Formatted at 2026-06-19 20:31:32 on dist-test-slave-f9rk"
server_key: "6b13f9bf4169820b47e82773585f8324"
server_key_iv: "cc86c92feaa32cfacf2184b65086007c"
server_key_version: "encryptionkey@0"
I20260619 20:31:32.935601 11794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:32.938787 11794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:32.940830 11794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:32.940943 11794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:32.944649 11794 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:36507
I20260619 20:31:32.944726 11861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:36507 every 8 connection(s)
I20260619 20:31:32.945055 11794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:32.945999 11862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:32.948192 11862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22: Bootstrap starting.
I20260619 20:31:32.948750 11862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:32.949043 11862 log.cc:824] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:32.949729 11862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22: No bootstrap required, opened a new log
I20260619 20:31:32.951359 11862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020ed78588d44814be7bd99423fc9e22" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36507 } }
I20260619 20:31:32.951519 11862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 20:31:32.951563 11862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020ed78588d44814be7bd99423fc9e22, State: Initialized, Role: FOLLOWER
I20260619 20:31:32.951757 11862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020ed78588d44814be7bd99423fc9e22" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36507 } }
I20260619 20:31:32.951848 11862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 20:31:32.951890 11862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 20:31:32.951939 11862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:32.952728 11862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020ed78588d44814be7bd99423fc9e22" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36507 } }
I20260619 20:31:32.952816 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 11794
I20260619 20:31:32.952862 11862 leader_election.cc:302] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 020ed78588d44814be7bd99423fc9e22; no voters:
I20260619 20:31:32.952950 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:32.953022 11862 leader_election.cc:288] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 20:31:32.953038 11865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:32.953151 11865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [term 1 LEADER]: Becoming Leader. State: Replica: 020ed78588d44814be7bd99423fc9e22, State: Running, Role: LEADER
I20260619 20:31:32.953150 22108 external_mini_cluster.cc:1428] Setting key 4139d3956b43a8216dc20d597275a90e
I20260619 20:31:32.953315 11862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:32.953295 11865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020ed78588d44814be7bd99423fc9e22" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36507 } }
I20260619 20:31:32.953882 11867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "020ed78588d44814be7bd99423fc9e22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020ed78588d44814be7bd99423fc9e22" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36507 } } }
I20260619 20:31:32.953984 11867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:32.953954 11866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 020ed78588d44814be7bd99423fc9e22. Latest consensus state: current_term: 1 leader_uuid: "020ed78588d44814be7bd99423fc9e22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020ed78588d44814be7bd99423fc9e22" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36507 } } }
I20260619 20:31:32.954035 11866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:32.954272 11876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:32.954824 11876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:32.955791 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:32.953679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52287 (local address 127.21.151.62:36507)
0619 20:31:32.953936 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:32.953957 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:32.953989 (+ 32us) server_negotiation.cc:407] Connection header received
0619 20:31:32.954102 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:32.954110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:32.954142 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:32.954548 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:32.955177 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:32.955181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:32.955181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:32.955186 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:32.955227 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:32.955240 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:32.955517 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:32.955552 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 20:31:32.955590 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":145,"threads_started":1}
I20260619 20:31:32.956859 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:32.956894 11876 catalog_manager.cc:1347] Generated new cluster ID: 0c4535a6844442adb2383c0bf8822f54
I20260619 20:31:32.956936 11876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:32.959324 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:32.958180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41737 (local address 127.21.151.62:36507)
0619 20:31:32.958213 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:32.958215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:32.958331 (+ 116us) server_negotiation.cc:407] Connection header received
0619 20:31:32.958395 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:32.958396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:32.958401 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:32.958458 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:32.959071 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:32.959072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:32.959073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 20:31:32.959077 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:32.959090 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 20:31:32.959094 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:32.959148 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:32.959231 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 20:31:32.959244 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 20:31:32.976023 11876 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:32.976214 11876 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:32.979851 11876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 020ed78588d44814be7bd99423fc9e22: Generated new TSK 0
I20260619 20:31:32.980039 11876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:32.984871 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 11794
2026-06-19T20:31:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1306 ms)
[----------] 18 tests from AuthTokenIssuingTest (23576 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:33 dist-test-slave-f9rk krb5kdc[11893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:33 dist-test-slave-f9rk krb5kdc[11893](info): set up 2 sockets
Jun 19 20:31:33 dist-test-slave-f9rk krb5kdc[11893](info): commencing operation
krb5kdc: starting...
W20260619 20:31:34.028913 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:34.144711 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:33463
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:33463,127.21.151.61:38701,127.21.151.60:33517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:34.225420 11906 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:31:34.225631 11906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:34.225648 11906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:34.225706 11906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:34.226814 11906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:34.226853 11906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:34.226867 11906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:34.226923 11906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:34.226941 11906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:34.228441 11906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:33463,127.21.151.61:38701,127.21.151.60:33517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:34.228703 11906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:34.228937 11906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:34.233847 11911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:34.233886 11914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:34.234030 11906 server_base.cc:1034] running on GCE node
W20260619 20:31:34.233911 11912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:34.234326 11906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:34.234617 11906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:34.235770 11906 hybrid_clock.cc:648] HybridClock initialized: now 1781901094235763 us; error 50 us; skew 500 ppm
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:34.238139 11906 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:34.238585 11906 webserver.cc:460] Webserver started at http://127.21.151.62:44227/ using document root <none> and password file <none>
I20260619 20:31:34.238806 11906 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:34.238870 11906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:34.239012 11906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:34.240134 11906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "3232c49687164413a75a8708f8c8b459"
format_stamp: "Formatted at 2026-06-19 20:31:34 on dist-test-slave-f9rk"
server_key: "e16778290aac666cf1cb8cd7b141c609"
server_key_iv: "80a82a718a13a3ab4c17aa06aee6b82a"
server_key_version: "encryptionkey@0"
I20260619 20:31:34.240515 11906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "3232c49687164413a75a8708f8c8b459"
format_stamp: "Formatted at 2026-06-19 20:31:34 on dist-test-slave-f9rk"
server_key: "e16778290aac666cf1cb8cd7b141c609"
server_key_iv: "80a82a718a13a3ab4c17aa06aee6b82a"
server_key_version: "encryptionkey@0"
I20260619 20:31:34.241782 11906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:34.242465 11921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:34.242630 11906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:34.242730 11906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "3232c49687164413a75a8708f8c8b459"
format_stamp: "Formatted at 2026-06-19 20:31:34 on dist-test-slave-f9rk"
server_key: "e16778290aac666cf1cb8cd7b141c609"
server_key_iv: "80a82a718a13a3ab4c17aa06aee6b82a"
server_key_version: "encryptionkey@0"
I20260619 20:31:34.242810 11906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:34.257295 11906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:34.259135 11906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:34.259306 11906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:34.263523 11906 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:33463
I20260619 20:31:34.263573 11973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:33463 every 8 connection(s)
I20260619 20:31:34.263960 11906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:34.264883 11974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:34.266336 11974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:31:34.269276 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 11906
I20260619 20:31:34.269397 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:34.269556 22108 external_mini_cluster.cc:1428] Setting key cb4d520320864c46dbe1a6fd9b6bec23
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:34.276919 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.267007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.21.151.62:33463)
0619 20:31:34.267202 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.267211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.267317 (+ 106us) server_negotiation.cc:407] Connection header received
0619 20:31:34.268006 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.268008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.268032 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.268128 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.268676 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.269826 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.270667 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.271066 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.271148 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.273675 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.273681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.273681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.273688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.275258 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.275762 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.275764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.275765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.275847 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.276110 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.276110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.276110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.276304 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.276361 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.276657 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.276751 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260619 20:31:34.277047 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.266935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33463 (local address 127.0.0.1:36868)
0619 20:31:34.267194 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.267238 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:34.267821 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:34.268133 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:34.268135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:34.268427 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:34.268573 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.268578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.269916 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.269917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.270574 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.270576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.271175 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.271177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.271262 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.271850 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:34.271858 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:34.273592 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:34.275361 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.275369 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.275371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.275684 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.275921 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.275922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.275922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.276037 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.276378 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:34.276386 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:34.276468 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 20:31:34.276602 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 20:31:34.276856 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":67,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.21.151.61@KRBTEST.COM; defaulting to no policy
I20260619 20:31:34.278035 11974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:31:34.278451 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.278305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38701 (local address 127.0.0.1:33122)
0619 20:31:34.278375 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.278432 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 20:31:34.278616 11923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38701: connect: Connection refused (error 111)
I20260619 20:31:34.278975 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.278891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38701 (local address 127.0.0.1:33130)
0619 20:31:34.278934 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.278958 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:31:34.279155 11974 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.61:38701: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38701: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab.
Entry for principal kudu/127.21.151.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab.
I20260619 20:31:34.293725 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.61:38701
--webserver_interface=127.21.151.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab
--principal=kudu/127.21.151.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:33463,127.21.151.61:38701,127.21.151.60:33517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:31:34.328378 11974 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } attempt: 1
I20260619 20:31:34.328933 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.328785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38701 (local address 127.0.0.1:33132)
0619 20:31:34.328849 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.328897 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 20:31:34.329310 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.329213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38701 (local address 127.0.0.1:33138)
0619 20:31:34.329260 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.329286 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 20:31:34.329473 11974 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.61:38701: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38701: connect: Connection refused (error 111)
W20260619 20:31:34.375584 11981 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:31:34.375842 11981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:34.375882 11981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:34.375944 11981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:34.377039 11981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:34.377103 11981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:34.377126 11981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:34.377156 11981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:34.377184 11981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:34.378711 11981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:33463,127.21.151.61:38701,127.21.151.60:33517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab
--principal=kudu/127.21.151.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.61:38701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.151.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:34.379009 11981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:34.379276 11981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:34.384032 11987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:34.384024 11986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:34.384024 11989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:34.384415 11981 server_base.cc:1034] running on GCE node
I20260619 20:31:34.384656 11981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:34.384908 11981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:34.386050 11981 hybrid_clock.cc:648] HybridClock initialized: now 1781901094386038 us; error 29 us; skew 500 ppm
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:34.388581 11981 init.cc:377] Logged in from keytab as kudu/127.21.151.61@KRBTEST.COM (short username kudu)
I20260619 20:31:34.389029 11981 webserver.cc:460] Webserver started at http://127.21.151.61:38277/ using document root <none> and password file <none>
I20260619 20:31:34.389262 11981 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:34.389339 11981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:34.389479 11981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:34.390985 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/data/instance:
uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3"
format_stamp: "Formatted at 2026-06-19 20:31:34 on dist-test-slave-f9rk"
server_key: "f0f79f8f086c041e30a4af2b4c3a2e3a"
server_key_iv: "34ad7b148246a94322b9cbc9c67650e6"
server_key_version: "encryptionkey@0"
I20260619 20:31:34.391348 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/wal/instance:
uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3"
format_stamp: "Formatted at 2026-06-19 20:31:34 on dist-test-slave-f9rk"
server_key: "f0f79f8f086c041e30a4af2b4c3a2e3a"
server_key_iv: "34ad7b148246a94322b9cbc9c67650e6"
server_key_version: "encryptionkey@0"
I20260619 20:31:34.392740 11981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:31:34.393455 11996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:34.393615 11981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:34.393720 11981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/wal
uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3"
format_stamp: "Formatted at 2026-06-19 20:31:34 on dist-test-slave-f9rk"
server_key: "f0f79f8f086c041e30a4af2b4c3a2e3a"
server_key_iv: "34ad7b148246a94322b9cbc9c67650e6"
server_key_version: "encryptionkey@0"
I20260619 20:31:34.393806 11981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:34.397619 11974 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } attempt: 2
I20260619 20:31:34.398020 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.397859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38701 (local address 127.0.0.1:33140)
0619 20:31:34.397941 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.397983 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 20:31:34.398332 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.398254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38701 (local address 127.0.0.1:33152)
0619 20:31:34.398295 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.398317 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 20:31:34.398557 11974 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.61:38701: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38701: connect: Connection refused (error 111)
I20260619 20:31:34.404341 11981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:34.406133 11981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:34.406291 11981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:34.410255 11981 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.61:38701
I20260619 20:31:34.410286 12048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.61:38701 every 8 connection(s)
I20260619 20:31:34.410646 11981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
I20260619 20:31:34.411582 12049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:34.413192 12049 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:31:34.418375 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 11981
I20260619 20:31:34.418493 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-1/wal/instance
I20260619 20:31:34.418681 22108 external_mini_cluster.cc:1428] Setting key daddb5a522462e341a8e850166100410
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:34.422842 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.413867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.21.151.62:33463)
0619 20:31:34.413933 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.413936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.414156 (+ 220us) server_negotiation.cc:407] Connection header received
0619 20:31:34.414584 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.414585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.414595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.414656 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.415258 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.415802 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.416752 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.416947 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.416999 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.419546 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.419548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.419548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.419556 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.421298 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.421794 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.421796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.421796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.421847 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.422137 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.422137 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.422138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.422247 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.422295 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.422642 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.422714 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:31:34.423089 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.413841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33463 (local address 127.0.0.1:36878)
0619 20:31:34.414024 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.414067 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:34.414506 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:34.414748 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:34.414751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:34.414996 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:34.415186 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.415191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.415880 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.415881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.416644 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.416645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.417027 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.417028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.417138 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.417754 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:34.417762 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:34.419454 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:34.421385 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.421392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.421394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.421718 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.421920 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.421921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.421921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.422074 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.422314 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:34.422320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:34.422503 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 20:31:34.422626 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 20:31:34.422809 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260619 20:31:34.423694 12049 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.151.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.60@KRBTEST.COM" created.
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:31:34.433586 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.423944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38701 (local address 127.0.0.1:33162)
0619 20:31:34.423995 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.424006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:34.424044 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:34.424380 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:34.424381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:34.424496 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:34.424553 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.424554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.425924 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.425925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.426431 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.426432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.427120 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.427121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.427172 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.427611 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:34.427647 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:34.430535 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:34.432348 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.432349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.432349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.432588 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.432794 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.432795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.432795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.432866 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.433123 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:34.433125 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:34.433175 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 20:31:34.433254 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:31:34.433368 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 20:31:34.433593 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.424020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.21.151.61:38701)
0619 20:31:34.424193 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.424199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.424223 (+ 24us) server_negotiation.cc:407] Connection header received
0619 20:31:34.424249 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.424252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.424267 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.424375 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.424642 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.425818 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.426521 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.427040 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.427125 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.430622 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.430630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.430631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.430638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.432286 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.432667 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.432670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.432670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.432726 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.432928 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.432928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.432929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.433055 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.433115 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.433296 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.433438 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260619 20:31:34.434321 12049 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:31:34.434679 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.434527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49232)
0619 20:31:34.434591 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.434646 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 20:31:34.434847 11999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
I20260619 20:31:34.435315 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.435180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49242)
0619 20:31:34.435238 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.435270 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 20:31:34.435472 12049 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:33517: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab.
Entry for principal kudu/127.21.151.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab.
I20260619 20:31:34.444725 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.60:33517
--webserver_interface=127.21.151.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab
--principal=kudu/127.21.151.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:33463,127.21.151.61:38701,127.21.151.60:33517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:31:34.484717 12049 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } attempt: 1
I20260619 20:31:34.485247 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.485090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49250)
0619 20:31:34.485164 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.485211 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 20:31:34.485610 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.485520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49260)
0619 20:31:34.485565 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.485591 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:31:34.485728 12049 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:33517: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
I20260619 20:31:34.489718 11974 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } attempt: 3
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:31:34.497210 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.490128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.21.151.61:38701)
0619 20:31:34.490166 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.490168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.490184 (+ 16us) server_negotiation.cc:407] Connection header received
0619 20:31:34.490240 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.490241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.490252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.490309 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.490685 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.491184 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.492221 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.492444 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.492499 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.494552 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.494554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.494554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.494561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.496068 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.496435 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.496436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.496437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.496494 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.496693 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.496694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.496694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.496810 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.496888 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.497040 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.497117 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:31:34.497241 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.490070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38701 (local address 127.0.0.1:33178)
0619 20:31:34.490126 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.490136 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:34.490178 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:34.490337 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:34.490338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:34.490530 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:34.490610 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.490613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.491267 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.491268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.492144 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.492147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.492522 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.492523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.492601 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.493073 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:34.493079 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:34.494459 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:34.496151 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.496153 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.496153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.496348 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.496562 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.496563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.496563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.496622 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.496879 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:34.496880 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:34.496928 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 20:31:34.497017 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 20:31:34.497117 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 20:31:34.497602 11974 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:31:34.497946 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.497843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49262)
0619 20:31:34.497892 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.497919 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 20:31:34.498225 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.498145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49266)
0619 20:31:34.498183 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.498205 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 20:31:34.498320 11974 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:33517: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
W20260619 20:31:34.527108 12056 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:31:34.527357 12056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:34.527387 12056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:34.527428 12056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:34.528576 12056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:34.528653 12056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:34.528678 12056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:34.528699 12056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:34.528728 12056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:34.529449 11974 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } attempt: 1
I20260619 20:31:34.529873 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.529726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49282)
0619 20:31:34.529809 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.529847 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 20:31:34.530234 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.530139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49284)
0619 20:31:34.530185 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.530209 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:31:34.530328 11974 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:33517: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
I20260619 20:31:34.530504 12056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:33463,127.21.151.61:38701,127.21.151.60:33517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab
--principal=kudu/127.21.151.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.60:33517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.151.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:34.530788 12056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:34.531032 12056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:34.535671 12061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:34.535708 12062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:34.535821 12056 server_base.cc:1034] running on GCE node
W20260619 20:31:34.535953 12064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:34.536193 12056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:34.536468 12056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:34.537608 12056 hybrid_clock.cc:648] HybridClock initialized: now 1781901094537597 us; error 27 us; skew 500 ppm
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:34.540096 12056 init.cc:377] Logged in from keytab as kudu/127.21.151.60@KRBTEST.COM (short username kudu)
I20260619 20:31:34.540491 12056 webserver.cc:460] Webserver started at http://127.21.151.60:43581/ using document root <none> and password file <none>
I20260619 20:31:34.540671 12056 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:34.540712 12056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:34.540800 12056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:34.541771 12056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/data/instance:
uuid: "834e559ca00642aa96688a4f575446af"
format_stamp: "Formatted at 2026-06-19 20:31:34 on dist-test-slave-f9rk"
server_key: "6c89f0e25ccc135ae01d5e3951a5f83c"
server_key_iv: "8bd3d1efc79df01fede633ac108d2835"
server_key_version: "encryptionkey@0"
I20260619 20:31:34.542055 12056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/wal/instance:
uuid: "834e559ca00642aa96688a4f575446af"
format_stamp: "Formatted at 2026-06-19 20:31:34 on dist-test-slave-f9rk"
server_key: "6c89f0e25ccc135ae01d5e3951a5f83c"
server_key_iv: "8bd3d1efc79df01fede633ac108d2835"
server_key_version: "encryptionkey@0"
I20260619 20:31:34.543268 12056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 20:31:34.543993 12071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:34.544162 12056 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:34.544272 12056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/wal
uuid: "834e559ca00642aa96688a4f575446af"
format_stamp: "Formatted at 2026-06-19 20:31:34 on dist-test-slave-f9rk"
server_key: "6c89f0e25ccc135ae01d5e3951a5f83c"
server_key_iv: "8bd3d1efc79df01fede633ac108d2835"
server_key_version: "encryptionkey@0"
I20260619 20:31:34.544348 12056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:34.551486 12056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:34.553663 12056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:34.553821 12056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:34.553939 12049 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } attempt: 2
I20260619 20:31:34.554389 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.554237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49298)
0619 20:31:34.554307 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.554349 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 20:31:34.554847 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.554723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49312)
0619 20:31:34.554784 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.554817 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 20:31:34.554960 12049 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:33517: Network error: Client connection negotiation failed: client connection to 127.21.151.60:33517: connect: Connection refused (error 111)
I20260619 20:31:34.557973 12056 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.60:33517
I20260619 20:31:34.558001 12123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.60:33517 every 8 connection(s)
I20260619 20:31:34.558337 12056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
I20260619 20:31:34.559363 12124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:34.559386 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 12056
I20260619 20:31:34.559494 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781900829246020-22108-0/minicluster-data/master-2/wal/instance
I20260619 20:31:34.559715 22108 external_mini_cluster.cc:1428] Setting key 46a3dac876e63970ca3774137b8fd216
I20260619 20:31:34.561795 12124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
I20260619 20:31:34.570904 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.562496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.21.151.62:33463)
0619 20:31:34.562566 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.562568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.562725 (+ 157us) server_negotiation.cc:407] Connection header received
0619 20:31:34.562792 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.562794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.562802 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.562850 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.563383 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.564110 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.565019 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.565313 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.565383 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.567791 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.567793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.567793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.567801 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.569306 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.569803 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.569805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.569806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.569867 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.570144 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.570145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.570145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.570312 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.570386 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.570708 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.570800 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 20:31:34.571049 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.562464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:33463 (local address 127.0.0.1:36890)
0619 20:31:34.562620 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.562658 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:34.562726 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:34.562892 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:34.562894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:34.563237 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:34.563309 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.563310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.564242 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.564243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.564920 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.564923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.565414 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.565415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.565518 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.566065 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:34.566076 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:34.567708 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:34.569433 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.569439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.569441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.569709 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.569951 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.569952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.569952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.570067 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.570391 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:34.570396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:34.570537 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 20:31:34.570657 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 20:31:34.570826 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260619 20:31:34.571487 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.560229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.21.151.60:33517)
0619 20:31:34.560385 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.560398 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.560434 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:31:34.560938 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.560945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.560992 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.561132 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.561910 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.563103 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.565037 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.565492 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.565548 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.568601 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.568609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.568609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.568617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.570153 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.570533 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.570536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.570536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.570594 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.570813 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.570814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.570814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.570966 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.571091 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.571291 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.571375 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260619 20:31:34.571898 12124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:34.579082 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.572199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38701 (local address 127.0.0.1:33182)
0619 20:31:34.572320 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.572330 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:34.572400 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:34.572546 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:34.572547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:34.572703 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:34.572766 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.572768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.573556 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.573557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.574251 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.574253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.574639 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.574640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.574689 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.575071 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:34.575076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:34.576368 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:34.578091 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.578092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.578093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.578325 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.578526 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.578527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.578527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.578593 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.578826 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:34.578827 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:34.578874 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 20:31:34.578943 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 20:31:34.579018 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 20:31:34.579288 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.572266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.21.151.61:38701)
0619 20:31:34.572318 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.572320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.572386 (+ 66us) server_negotiation.cc:407] Connection header received
0619 20:31:34.572461 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.572462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.572475 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.572539 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.572842 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.573479 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.574334 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.574556 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.574626 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.576460 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.576461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.576461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.576469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.577997 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.578404 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.578405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.578405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.578456 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.578658 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.578658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.578659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.578761 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.578811 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.578994 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.579069 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:31:34.579681 12124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:31:34.580145 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.572428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.21.151.62:33463)
0619 20:31:34.572466 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.572468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.572475 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:31:34.572494 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.572494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.572503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.572553 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.573104 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.573721 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.574575 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.574830 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.574893 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.577244 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.577246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.577246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.577254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.578780 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.579251 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.579252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.579252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.579314 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.579539 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.579540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.579540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.579724 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.579924 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.579960 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.580038 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:31:34.586313 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.579965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49342)
0619 20:31:34.580007 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.580016 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:34.580052 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:34.580168 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:34.580169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:34.580305 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:34.580335 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.580336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.581112 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.581113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.581587 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.581588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.581981 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.581981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.582029 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.582406 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:34.582410 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:34.583471 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:34.585212 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.585213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.585213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.585433 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.585650 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.585651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.585651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.585729 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.585987 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:34.585987 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:34.586024 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 20:31:34.586088 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 20:31:34.586186 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 20:31:34.586320 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.580001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.21.151.60:33517)
0619 20:31:34.580038 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.580040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.580048 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:31:34.580106 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.580107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.580116 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.580157 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.580435 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.581023 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.581679 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.581898 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.581983 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.583570 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.583571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.583571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.583578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.585128 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.585513 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.585514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.585515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.585570 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.585813 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.585813 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.585814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.585912 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.585993 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.586123 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.586236 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":22}
I20260619 20:31:34.587867 12124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af: Bootstrap starting.
I20260619 20:31:34.588315 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.580899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.21.151.61:38701)
0619 20:31:34.580940 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.580942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.580950 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:31:34.580972 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.580973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.580982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.581024 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.581482 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.581929 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.582752 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.582981 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.583068 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.585727 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.585729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.585729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.585736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.587201 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.587556 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.587557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.587557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.587611 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.587818 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.587818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.587819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.587961 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.588116 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.588153 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.588219 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 20:31:34.588915 12124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:34.589241 12124 log.cc:824] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:34.589919 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:34.590152 12124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af: No bootstrap required, opened a new log
I20260619 20:31:34.591096 11948 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3232c49687164413a75a8708f8c8b459"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36898
I20260619 20:31:34.591846 12124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } }
I20260619 20:31:34.592011 12124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834e559ca00642aa96688a4f575446af, State: Initialized, Role: FOLLOWER
I20260619 20:31:34.592103 12124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } }
I20260619 20:31:34.592420 12124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:34.592398 12130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } } }
I20260619 20:31:34.592499 12130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:31:34.594223 12141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:31:34.594305 12141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:31:34.594558 12056 master_runner.cc:186] Error getting master registration for 127.21.151.62:33463: OK, instance_id { permanent_uuid: "3232c49687164413a75a8708f8c8b459" instance_seqno: 1781901094262346 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:31:34.596987 12056 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:31:34.605460 11974 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } attempt: 2
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
I20260619 20:31:34.612584 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.605769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49352)
0619 20:31:34.605816 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.605828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:34.605879 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:34.606064 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:34.606065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:34.606244 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:34.606287 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.606289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.606931 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.606932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.607450 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.607452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.607882 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.607883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.607940 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.608416 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:34.608422 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:34.609811 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:34.611431 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.611432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.611432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.611681 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.611885 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.611886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.611887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.611952 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.612255 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:34.612256 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:34.612306 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 20:31:34.612381 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 20:31:34.612483 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 20:31:34.612586 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.605829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.21.151.60:33517)
0619 20:31:34.605910 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.605912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.605922 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:31:34.605974 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.605975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.605984 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.606060 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.606368 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.606830 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.607537 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.607798 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.607882 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.609919 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.609920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.609920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.609928 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.611333 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.611751 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.611752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.611752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.611812 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.612013 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.612014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.612014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.612186 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.612251 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.612414 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.612493 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 20:31:34.614070 11974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459: Bootstrap starting.
I20260619 20:31:34.614698 11974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:34.614996 11974 log.cc:824] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:34.615806 11974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459: No bootstrap required, opened a new log
I20260619 20:31:34.616806 11974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } }
I20260619 20:31:34.616947 11974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3232c49687164413a75a8708f8c8b459, State: Initialized, Role: FOLLOWER
I20260619 20:31:34.617041 11974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } }
I20260619 20:31:34.617334 12144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } } }
I20260619 20:31:34.617385 11974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:34.617419 12144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:34.617525 11948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 20:31:34.617606 11948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:34.618381 11948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } }
I20260619 20:31:34.618702 11948 leader_election.cc:288] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [CANDIDATE]: Term 1 election: Requested vote from peers 1013e4d167504fb18c6cdfe1f8c4e2b3 (127.21.151.61:38701), 834e559ca00642aa96688a4f575446af (127.21.151.60:33517)
I20260619 20:31:34.619038 12098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3232c49687164413a75a8708f8c8b459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "834e559ca00642aa96688a4f575446af"
I20260619 20:31:34.619194 12098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:34.620350 12098 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3232c49687164413a75a8708f8c8b459 in term 1.
I20260619 20:31:34.620652 11924 leader_election.cc:302] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3232c49687164413a75a8708f8c8b459, 834e559ca00642aa96688a4f575446af; no voters:
I20260619 20:31:34.620772 12144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:34.620905 11906 master_runner.cc:186] Error getting master registration for 127.21.151.61:38701: OK, instance_id { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" instance_seqno: 1781901094409172 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:31:34.623596 11906 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:31:34.620909 12144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [term 1 LEADER]: Becoming Leader. State: Replica: 3232c49687164413a75a8708f8c8b459, State: Running, Role: LEADER
I20260619 20:31:34.623845 12144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } }
I20260619 20:31:34.624241 12144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3232c49687164413a75a8708f8c8b459. Latest consensus state: current_term: 1 leader_uuid: "3232c49687164413a75a8708f8c8b459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } } }
I20260619 20:31:34.624334 12144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [sys.catalog]: This master's current role is: LEADER
W20260619 20:31:34.625479 12155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:31:34.625547 12155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:31:34.625720 12165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:34.625943 12165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:34.629078 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.620601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.21.151.61:38701)
0619 20:31:34.620644 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.620646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.621109 (+ 463us) server_negotiation.cc:407] Connection header received
0619 20:31:34.621209 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.621210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.621222 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.621248 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.621896 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.622706 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.623434 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.623745 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.623808 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.625373 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.625375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.625375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.625383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.627257 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.627868 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.627870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.627870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.627946 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.628275 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.628277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.628277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.628437 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.628518 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.628893 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.628967 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:31:34.630705 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.620538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.21.151.60:33517)
0619 20:31:34.620741 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.620743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.622003 (+ 1260us) server_negotiation.cc:407] Connection header received
0619 20:31:34.622090 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.622091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.622100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.622129 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.623473 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.624147 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.625414 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.625705 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.625748 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.627539 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.627540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.627540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.627547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.629284 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.629889 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.629890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.629891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.629957 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.630156 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.630157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.630157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.630271 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.630333 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.630536 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.630611 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260619 20:31:34.632453 12098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [term 1 FOLLOWER]: Refusing update from remote peer 3232c49687164413a75a8708f8c8b459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:34.632781 12161 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:34.634089 12130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3232c49687164413a75a8708f8c8b459. Latest consensus state: current_term: 1 leader_uuid: "3232c49687164413a75a8708f8c8b459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } } }
I20260619 20:31:34.634181 12130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:34.640163 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.620293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36902 (local address 127.21.151.62:33463)
0619 20:31:34.620333 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.620335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.627758 (+ 7423us) server_negotiation.cc:407] Connection header received
0619 20:31:34.627856 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.627857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.627867 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.627894 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.628504 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.629157 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.630093 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.630383 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.630457 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.637148 (+ 6691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.637149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.637150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.637157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.638778 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.639232 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.639234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.639234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.639312 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.639542 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.639543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.639543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.639713 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.639803 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.639981 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.640055 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:31:34.646087 12049 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } attempt: 3
I20260619 20:31:34.648828 12167 mvcc.cc:205] Tried to move back new op lower bound from 7298666883612221440 to 7298666883579924480. Current Snapshot: MvccSnapshot[applied={T|T < 7298666883612221440}]
I20260619 20:31:34.649245 12144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3232c49687164413a75a8708f8c8b459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } } }
I20260619 20:31:34.649351 12144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:34.649955 12130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3232c49687164413a75a8708f8c8b459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } } }
I20260619 20:31:34.650048 12130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:34.650434 12168 mvcc.cc:205] Tried to move back new op lower bound from 7298666883612221440 to 7298666883579924480. Current Snapshot: MvccSnapshot[applied={T|T < 7298666883612221440}]
Jun 19 20:31:34 dist-test-slave-f9rk krb5kdc[11893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901094, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
I20260619 20:31:34.651603 12165 catalog_manager.cc:1347] Generated new cluster ID: 583b7bc519fc4cd38529421543c723b6
I20260619 20:31:34.651695 12165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:34.655212 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.646381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:33517 (local address 127.0.0.1:49366)
0619 20:31:34.646435 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:34.646445 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:34.646485 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:34.647015 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:34.647016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:34.647227 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:34.647285 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.647287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.648312 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.648313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.649196 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:34.649199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.649814 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.649815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:34.649874 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.650446 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:34.650452 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:34.652015 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:34.653896 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.653898 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.653898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.654134 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.654349 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:34.654349 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:34.654350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:34.654410 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:34.654877 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:34.654878 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:34.654931 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 20:31:34.654990 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 20:31:34.655101 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 20:31:34.655593 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:34.646445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.21.151.60:33517)
0619 20:31:34.646911 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.646913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.646921 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:31:34.646942 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.646943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.646953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.647006 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.647369 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.648204 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.649351 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:34.649659 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:34.649723 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:34.652115 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:34.652117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:34.652117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:34.652124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:34.653794 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.654218 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.654219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.654220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.654278 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:34.654472 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:34.654473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:34.654473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:34.654602 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:34.655380 (+ 778us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:34.655425 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 20:31:34.655504 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
I20260619 20:31:34.657210 12049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3: Bootstrap starting.
I20260619 20:31:34.657960 12049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:34.658255 12049 log.cc:824] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:34.659123 12049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3: No bootstrap required, opened a new log
I20260619 20:31:34.661166 12049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } }
I20260619 20:31:34.661305 12049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1013e4d167504fb18c6cdfe1f8c4e2b3, State: Initialized, Role: FOLLOWER
I20260619 20:31:34.661381 12049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } }
I20260619 20:31:34.661680 12049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:34.661798 12022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:34.662719 12022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [term 1 FOLLOWER]: Refusing update from remote peer 3232c49687164413a75a8708f8c8b459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:34.663038 12023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3232c49687164413a75a8708f8c8b459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3"
I20260619 20:31:34.663232 12171 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 20:31:34.663657 12185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:31:34.663724 12185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:31:34.664517 12023 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3232c49687164413a75a8708f8c8b459 in term 1.
I20260619 20:31:34.666589 12174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3232c49687164413a75a8708f8c8b459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } } }
I20260619 20:31:34.666675 12174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:34.666729 12174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3232c49687164413a75a8708f8c8b459. Latest consensus state: current_term: 1 leader_uuid: "3232c49687164413a75a8708f8c8b459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } } }
I20260619 20:31:34.666777 12174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:34.666812 12174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3232c49687164413a75a8708f8c8b459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } } }
I20260619 20:31:34.666872 12174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:34.670912 12165 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:34.671675 12165 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:34.676690 12161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3232c49687164413a75a8708f8c8b459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3232c49687164413a75a8708f8c8b459" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 33463 } } peers { permanent_uuid: "1013e4d167504fb18c6cdfe1f8c4e2b3" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38701 } } peers { permanent_uuid: "834e559ca00642aa96688a4f575446af" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 33517 } } }
I20260619 20:31:34.676841 12161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:34.677904 12165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3232c49687164413a75a8708f8c8b459: Generated new TSK 0
I20260619 20:31:34.678342 12165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 20:31:34.680250 12188 mvcc.cc:205] Tried to move back new op lower bound from 7298666883612221440 to 7298666883579924480. Current Snapshot: MvccSnapshot[applied={T|T < 7298666883612221440}]
W20260619 20:31:34.702922 11977 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.701250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36904 (local address 127.21.151.62:33463)
0619 20:31:34.701295 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.701298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.702203 (+ 905us) server_negotiation.cc:407] Connection header received
0619 20:31:34.702312 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.702313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.702323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.702355 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.702844 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36904: BlockingRecv error: recv got EOF from 127.0.0.1:36904 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 20:31:34.703718 12127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.701623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49368 (local address 127.21.151.60:33517)
0619 20:31:34.701664 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.701666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.703160 (+ 1494us) server_negotiation.cc:407] Connection header received
0619 20:31:34.703256 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.703257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.703267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.703295 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.703634 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49368: BlockingRecv error: recv got EOF from 127.0.0.1:49368 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:31:34.704520 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.701948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.21.151.61:38701)
0619 20:31:34.701986 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.701987 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.703918 (+ 1931us) server_negotiation.cc:407] Connection header received
0619 20:31:34.704091 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.704092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.704102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.704129 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.704458 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33222: BlockingRecv error: recv got EOF from 127.0.0.1:33222 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 20:31:34.706878 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.706145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.21.151.61:38701)
0619 20:31:34.706179 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.706181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.706375 (+ 194us) server_negotiation.cc:407] Connection header received
0619 20:31:34.706496 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.706497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.706515 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.706543 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.706833 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33226: BlockingRecv error: recv got EOF from 127.0.0.1:33226 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 20:31:34.709832 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.709068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.21.151.61:38701)
0619 20:31:34.709118 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.709120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.709344 (+ 224us) server_negotiation.cc:407] Connection header received
0619 20:31:34.709451 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.709452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.709461 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.709489 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.709789 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33234: BlockingRecv error: recv got EOF from 127.0.0.1:33234 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 20:31:34.713663 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.712928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.21.151.61:38701)
0619 20:31:34.712961 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.712962 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.713175 (+ 213us) server_negotiation.cc:407] Connection header received
0619 20:31:34.713287 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.713288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.713298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.713326 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.713620 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33242: BlockingRecv error: recv got EOF from 127.0.0.1:33242 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 20:31:34.719763 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.718854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.21.151.61:38701)
0619 20:31:34.718896 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.718897 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.719136 (+ 239us) server_negotiation.cc:407] Connection header received
0619 20:31:34.719235 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.719236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.719245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.719273 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.719583 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33246: BlockingRecv error: recv got EOF from 127.0.0.1:33246 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 20:31:34.729921 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.729034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.21.151.61:38701)
0619 20:31:34.729075 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.729077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.729355 (+ 278us) server_negotiation.cc:407] Connection header received
0619 20:31:34.729455 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.729456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.729467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.729495 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.729864 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33256: BlockingRecv error: recv got EOF from 127.0.0.1:33256 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 20:31:34.748469 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.747367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.21.151.61:38701)
0619 20:31:34.747411 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.747414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.747741 (+ 327us) server_negotiation.cc:407] Connection header received
0619 20:31:34.747856 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.747857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.747869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.747901 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.748410 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33268: BlockingRecv error: recv got EOF from 127.0.0.1:33268 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:31:34.782732 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.781845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.21.151.61:38701)
0619 20:31:34.781889 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.781891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.782144 (+ 253us) server_negotiation.cc:407] Connection header received
0619 20:31:34.782247 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.782248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.782260 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.782290 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.782671 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33280: BlockingRecv error: recv got EOF from 127.0.0.1:33280 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:31:34.849021 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.848232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.21.151.61:38701)
0619 20:31:34.848297 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.848299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.848431 (+ 132us) server_negotiation.cc:407] Connection header received
0619 20:31:34.848507 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.848508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.848520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.848602 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.848966 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33292: BlockingRecv error: recv got EOF from 127.0.0.1:33292 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 20:31:34.978993 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:34.978388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.21.151.61:38701)
0619 20:31:34.978461 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:34.978463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:34.978498 (+ 35us) server_negotiation.cc:407] Connection header received
0619 20:31:34.978566 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:34.978567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:34.978577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:34.978641 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:34.978944 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33296: BlockingRecv error: recv got EOF from 127.0.0.1:33296 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 20:31:35.237066 12053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:35.236445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.21.151.61:38701)
0619 20:31:35.236524 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:35.236526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:35.236536 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:31:35.236605 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:35.236606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:35.236615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:35.236667 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:35.236986 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33308: BlockingRecv error: recv got EOF from 127.0.0.1:33308 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 20:31:35.594754 12141 catalog_manager.cc:1259] Loaded cluster ID: 583b7bc519fc4cd38529421543c723b6
I20260619 20:31:35.594852 12141 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af: loading cluster ID for follower catalog manager: success
I20260619 20:31:35.595662 12141 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af: acquiring CA information for follower catalog manager: success
I20260619 20:31:35.596004 12141 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 834e559ca00642aa96688a4f575446af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:31:35.664156 12185 catalog_manager.cc:1259] Loaded cluster ID: 583b7bc519fc4cd38529421543c723b6
I20260619 20:31:35.664234 12185 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3: loading cluster ID for follower catalog manager: success
I20260619 20:31:35.665150 12185 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3: acquiring CA information for follower catalog manager: success
I20260619 20:31:35.665488 12185 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1013e4d167504fb18c6cdfe1f8c4e2b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:31:35.752678 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:35.750256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.21.151.61:38701)
0619 20:31:35.750375 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:35.750377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:35.750387 (+ 10us) server_negotiation.cc:407] Connection header received
0619 20:31:35.750422 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:35.750423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:35.750472 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:35.750556 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:35.750681 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:35.751161 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:35.751993 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:35.752189 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:35.752245 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:35.752382 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:35.752491 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:35.752551 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:35.752588 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 20:31:35.752607 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260619 20:31:35.756960 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:35.754539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.21.151.60:33517)
0619 20:31:35.754637 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:35.754640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:35.754651 (+ 11us) server_negotiation.cc:407] Connection header received
0619 20:31:35.754695 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:35.754696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:35.754739 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:35.754838 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:35.754930 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:35.755409 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:35.756182 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:35.756396 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:35.756481 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:35.756635 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:35.756740 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:35.756827 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:35.756850 (+ 23us) server_negotiation.cc:299] Negotiation successful
0619 20:31:35.756869 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":43,"threads_started":1}
I20260619 20:31:35.761643 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:35.758707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.21.151.61:38701)
0619 20:31:35.758744 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:35.758746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:35.758831 (+ 85us) server_negotiation.cc:407] Connection header received
0619 20:31:35.758892 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:35.758893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:35.758905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:35.758953 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:35.759102 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:35.759860 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:35.760725 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:35.760989 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:35.761053 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:35.761348 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:35.761381 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:35.761463 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:35.761525 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 20:31:35.761549 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:31:35.761744 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:35.758658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.21.151.60:33517)
0619 20:31:35.758709 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:35.758711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:35.758842 (+ 131us) server_negotiation.cc:407] Connection header received
0619 20:31:35.758962 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:35.758963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:35.758974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:35.759025 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:35.759166 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:35.759869 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:35.760908 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:35.761173 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:35.761280 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:35.761484 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:35.761513 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:35.761543 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:35.761652 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 20:31:35.761673 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 20:31:35.761749 12272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:35.758558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.21.151.62:33463)
0619 20:31:35.758722 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:35.758724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:35.758733 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:31:35.758824 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:35.758826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:35.758879 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:35.758954 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:35.759145 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:35.759675 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:35.760739 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:35.761054 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:35.761096 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:35.761374 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:35.761491 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:35.761549 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:35.761651 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 20:31:35.761674 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":55,"threads_started":1}
I20260619 20:31:35.763713 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 11906
I20260619 20:31:35.769948 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 11981
I20260619 20:31:35.775305 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 12056
2026-06-19T20:31:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2792 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 20:31:35 dist-test-slave-f9rk krb5kdc[12276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 20:31:35 dist-test-slave-f9rk krb5kdc[12276](info): set up 2 sockets
Jun 19 20:31:35 dist-test-slave-f9rk krb5kdc[12276](info): commencing operation
krb5kdc: starting...
W20260619 20:31:36.797132 22108 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/test-user.keytab.
Jun 19 20:31:36 dist-test-slave-f9rk krb5kdc[12276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T20:31:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T20:31:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.151.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
Entry for principal kudu/127.21.151.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab.
I20260619 20:31:36.912729 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:36493
--webserver_interface=127.21.151.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:36493,127.21.151.61:38335,127.21.151.60:38975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 20:31:36.994799 12289 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:31:36.995020 12289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:36.995038 12289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:36.995057 12289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:36.996366 12289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:36.996410 12289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:36.996424 12289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:36.996438 12289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:36.996495 12289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:36.998044 12289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:36493,127.21.151.61:38335,127.21.151.60:38975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.62.keytab
--principal=kudu/127.21.151.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.62:36493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.151.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:36.998287 12289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:36.998514 12289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:37.003500 12297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:37.003552 12289 server_base.cc:1034] running on GCE node
W20260619 20:31:37.003500 12294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:37.003506 12295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:37.004040 12289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:37.004387 12289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:37.005566 12289 hybrid_clock.cc:648] HybridClock initialized: now 1781901097005542 us; error 38 us; skew 500 ppm
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:37.012265 12289 init.cc:377] Logged in from keytab as kudu/127.21.151.62@KRBTEST.COM (short username kudu)
I20260619 20:31:37.012774 12289 webserver.cc:460] Webserver started at http://127.21.151.62:33673/ using document root <none> and password file <none>
I20260619 20:31:37.013016 12289 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:37.013091 12289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:37.013242 12289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:37.014343 12289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/data/instance:
uuid: "c8f2fa3b0be844ef8efaab96579f0b62"
format_stamp: "Formatted at 2026-06-19 20:31:37 on dist-test-slave-f9rk"
server_key: "d21ef42366dcfe22234c2443237dd1a6"
server_key_iv: "6fb4108414330da3e16a1ed58cd7f8ee"
server_key_version: "encryptionkey@0"
I20260619 20:31:37.014708 12289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/wal/instance:
uuid: "c8f2fa3b0be844ef8efaab96579f0b62"
format_stamp: "Formatted at 2026-06-19 20:31:37 on dist-test-slave-f9rk"
server_key: "d21ef42366dcfe22234c2443237dd1a6"
server_key_iv: "6fb4108414330da3e16a1ed58cd7f8ee"
server_key_version: "encryptionkey@0"
I20260619 20:31:37.016144 12289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:31:37.016855 12304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:37.017051 12289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:37.017190 12289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/wal
uuid: "c8f2fa3b0be844ef8efaab96579f0b62"
format_stamp: "Formatted at 2026-06-19 20:31:37 on dist-test-slave-f9rk"
server_key: "d21ef42366dcfe22234c2443237dd1a6"
server_key_iv: "6fb4108414330da3e16a1ed58cd7f8ee"
server_key_version: "encryptionkey@0"
I20260619 20:31:37.017309 12289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:37.025884 12289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:37.027788 12289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:37.027952 12289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:37.032008 12289 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.62:36493
I20260619 20:31:37.032040 12356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.62:36493 every 8 connection(s)
I20260619 20:31:37.032406 12289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/data/info.pb
I20260619 20:31:37.033356 12357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:37.034835 12357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:31:37.037575 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 12289
I20260619 20:31:37.037647 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-0/wal/instance
I20260619 20:31:37.037859 22108 external_mini_cluster.cc:1428] Setting key f834de094cf6d40809660e690957fb8c
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:37.044499 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.035469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:36493 (local address 127.0.0.1:39968)
0619 20:31:37.035661 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.035707 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:37.036147 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:37.036396 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:37.036398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:37.036641 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:37.036819 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.036822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.037854 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.037855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.038580 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.038582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.039128 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.039130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.039194 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.039736 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:37.039745 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:37.041078 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:37.042884 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.042891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.042893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.043082 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.043330 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.043331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.043332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.043422 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.043837 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:37.043842 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:37.043902 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 20:31:37.044012 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 20:31:37.044251 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":94,"thread_start_us":72,"threads_started":1}
I20260619 20:31:37.044502 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.035679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.21.151.62:36493)
0619 20:31:37.035796 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.035801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.035837 (+ 36us) server_negotiation.cc:407] Connection header received
0619 20:31:37.036273 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.036275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.036290 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.036367 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.036903 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.037759 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.038685 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.039001 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.039151 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.041179 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.041186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.041186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.041193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.042782 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.043178 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.043181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.043181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.043259 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.043490 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.043491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.043491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.043765 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.043859 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.044044 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.044173 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for kudu/127.21.151.61@KRBTEST.COM; defaulting to no policy
I20260619 20:31:37.045322 12357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:31:37.045761 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.045622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38335 (local address 127.0.0.1:41566)
0619 20:31:37.045685 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.045735 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 20:31:37.045903 12305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38335: connect: Connection refused (error 111)
I20260619 20:31:37.046332 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.046219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38335 (local address 127.0.0.1:41568)
0619 20:31:37.046265 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.046289 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 20:31:37.046433 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.61:38335: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38335: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab.
Entry for principal kudu/127.21.151.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab.
I20260619 20:31:37.062119 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.61:38335
--webserver_interface=127.21.151.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab
--principal=kudu/127.21.151.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:36493,127.21.151.61:38335,127.21.151.60:38975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:31:37.095762 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } attempt: 1
I20260619 20:31:37.096309 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.096159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38335 (local address 127.0.0.1:41576)
0619 20:31:37.096240 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.096281 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 20:31:37.096676 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.096566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38335 (local address 127.0.0.1:41580)
0619 20:31:37.096614 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.096639 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 20:31:37.096786 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.61:38335: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38335: connect: Connection refused (error 111)
W20260619 20:31:37.145970 12364 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:31:37.146236 12364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:37.146270 12364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:37.146291 12364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:37.147395 12364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:37.147429 12364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:37.147444 12364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:37.147456 12364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:37.147468 12364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:37.148882 12364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:36493,127.21.151.61:38335,127.21.151.60:38975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.61.keytab
--principal=kudu/127.21.151.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.61:38335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.151.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:37.149065 12364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:37.149225 12364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:37.154121 12372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:37.154132 12370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:37.154292 12364 server_base.cc:1034] running on GCE node
W20260619 20:31:37.154210 12369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:37.154606 12364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:37.154877 12364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:37.156021 12364 hybrid_clock.cc:648] HybridClock initialized: now 1781901097155998 us; error 42 us; skew 500 ppm
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:37.158380 12364 init.cc:377] Logged in from keytab as kudu/127.21.151.61@KRBTEST.COM (short username kudu)
I20260619 20:31:37.158768 12364 webserver.cc:460] Webserver started at http://127.21.151.61:34475/ using document root <none> and password file <none>
I20260619 20:31:37.158949 12364 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:37.158988 12364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:37.159075 12364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:37.160167 12364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/data/instance:
uuid: "3a33503a99564da18cf567d1f5a59a07"
format_stamp: "Formatted at 2026-06-19 20:31:37 on dist-test-slave-f9rk"
server_key: "34da089157bbf32b990f2cd14cc7bb54"
server_key_iv: "3b1cda1b3522e78e022586b0f2e41a6a"
server_key_version: "encryptionkey@0"
I20260619 20:31:37.160573 12364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/wal/instance:
uuid: "3a33503a99564da18cf567d1f5a59a07"
format_stamp: "Formatted at 2026-06-19 20:31:37 on dist-test-slave-f9rk"
server_key: "34da089157bbf32b990f2cd14cc7bb54"
server_key_iv: "3b1cda1b3522e78e022586b0f2e41a6a"
server_key_version: "encryptionkey@0"
I20260619 20:31:37.161883 12364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 20:31:37.162569 12379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:37.162750 12364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 20:31:37.162859 12364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/wal
uuid: "3a33503a99564da18cf567d1f5a59a07"
format_stamp: "Formatted at 2026-06-19 20:31:37 on dist-test-slave-f9rk"
server_key: "34da089157bbf32b990f2cd14cc7bb54"
server_key_iv: "3b1cda1b3522e78e022586b0f2e41a6a"
server_key_version: "encryptionkey@0"
I20260619 20:31:37.162937 12364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:37.164918 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } attempt: 2
I20260619 20:31:37.165303 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.165176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38335 (local address 127.0.0.1:41586)
0619 20:31:37.165234 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.165278 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 20:31:37.165603 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.165513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38335 (local address 127.0.0.1:41594)
0619 20:31:37.165557 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.165580 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 20:31:37.165686 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.61:38335: Network error: Client connection negotiation failed: client connection to 127.21.151.61:38335: connect: Connection refused (error 111)
I20260619 20:31:37.180074 12364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:37.181954 12364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:37.182114 12364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:37.186267 12364 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.61:38335
I20260619 20:31:37.186332 12431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.61:38335 every 8 connection(s)
I20260619 20:31:37.186679 12364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/data/info.pb
I20260619 20:31:37.186852 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 12364
I20260619 20:31:37.186960 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-1/wal/instance
I20260619 20:31:37.187173 22108 external_mini_cluster.cc:1428] Setting key 1ef022bb7d91d901b32506fb66ed917e
I20260619 20:31:37.187784 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:37.189302 12432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.151.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.151.60@KRBTEST.COM" created.
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:37.200841 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.190054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.21.151.62:36493)
0619 20:31:37.190132 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.190134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.190318 (+ 184us) server_negotiation.cc:407] Connection header received
0619 20:31:37.190835 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.190836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.190850 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.190901 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.191524 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.192079 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.193135 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.193345 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.193441 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.197508 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.197510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.197511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.197520 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.199256 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.199805 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.199806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.199806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.199872 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.200171 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.200172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.200172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.200282 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.200363 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.200656 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.200745 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 20:31:37.201026 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.190022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:36493 (local address 127.0.0.1:39984)
0619 20:31:37.190198 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.190233 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:37.190717 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:37.190978 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:37.190981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:37.191228 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:37.191419 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.191423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.192195 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.192196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.193022 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.193024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.193479 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.193480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.193601 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.194185 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:37.194193 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:37.197391 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:37.199342 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.199349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.199351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.199708 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.199948 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.199949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.199949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.200101 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.200357 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:37.200364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:37.200512 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 20:31:37.200623 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 20:31:37.200787 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260619 20:31:37.201802 12432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:31:37.210031 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.202088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38335 (local address 127.0.0.1:41604)
0619 20:31:37.202284 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.202295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:37.202346 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:37.202507 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:37.202508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:37.202641 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:37.202688 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.202690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.203548 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.203549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.204113 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.204115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.204620 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.204621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.204685 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.205129 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:37.205133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:37.207123 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:37.208980 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.208980 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.208981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.209134 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.209351 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.209352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.209352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.209404 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.209684 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:37.209685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:37.209724 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 20:31:37.209796 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 20:31:37.209892 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260619 20:31:37.210055 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.202152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.21.151.61:38335)
0619 20:31:37.202295 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.202301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.202344 (+ 43us) server_negotiation.cc:407] Connection header received
0619 20:31:37.202409 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.202412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.202424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.202508 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.202764 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.203448 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.204209 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.204526 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.204619 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.207208 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.207216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.207216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.207223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.208856 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.209228 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.209231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.209231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.209284 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.209474 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.209475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.209476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.209611 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.209705 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.209825 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.209943 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260619 20:31:37.210700 12432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:31:37.211045 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.210924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:58946)
0619 20:31:37.210973 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.211016 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.151.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab.
Entry for principal kudu/127.21.151.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab.
W20260619 20:31:37.211174 12381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
I20260619 20:31:37.211585 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.211488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:58958)
0619 20:31:37.211540 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.211563 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 20:31:37.211726 12432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:38975: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
I20260619 20:31:37.211946 22108 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM6qw9r/build/release/bin/kudu
/tmp/dist-test-taskM6qw9r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.60:38975
--webserver_interface=127.21.151.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab
--principal=kudu/127.21.151.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.151.20:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.151.62:36493,127.21.151.61:38335,127.21.151.60:38975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 20:31:37.256831 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } attempt: 3
I20260619 20:31:37.260959 12432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } attempt: 1
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:31:37.261335 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.261230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:58964)
0619 20:31:37.261272 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.261307 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 20:31:37.261785 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.261635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:58966)
0619 20:31:37.261709 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.261746 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 20:31:37.262182 12432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:38975: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
I20260619 20:31:37.264366 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.257297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.21.151.61:38335)
0619 20:31:37.257372 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.257374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.257387 (+ 13us) server_negotiation.cc:407] Connection header received
0619 20:31:37.257436 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.257437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.257454 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.257516 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.257850 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.258338 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.259101 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.259330 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.259438 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.261606 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.261607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.261608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.261615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.263191 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.263582 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.263583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.263583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.263681 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.263923 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.263924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.263924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.264030 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.264083 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.264250 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.264311 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 20:31:37.264427 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.257224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38335 (local address 127.0.0.1:41608)
0619 20:31:37.257276 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.257289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:37.257364 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:37.257524 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:37.257525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:37.257694 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:37.257759 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.257761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.258450 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.258452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.259005 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.259008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.259446 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.259448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.259532 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.260106 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:37.260111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:37.261504 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:37.263278 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.263279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.263280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.263500 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.263750 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.263775 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.263776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.263857 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.264096 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:37.264097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:37.264154 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 20:31:37.264227 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 20:31:37.264333 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 20:31:37.264858 12357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } has no permanent_uuid. Determining permanent_uuid...
I20260619 20:31:37.265239 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.265123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:58980)
0619 20:31:37.265177 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.265214 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:31:37.265616 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.265495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:58990)
0619 20:31:37.265562 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.265587 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 20:31:37.265741 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:38975: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
W20260619 20:31:37.294713 12439 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 20:31:37.294935 12439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 20:31:37.294952 12439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 20:31:37.294972 12439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 20:31:37.296147 12439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 20:31:37.296186 12439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 20:31:37.296200 12439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 20:31:37.296213 12439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 20:31:37.296226 12439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 20:31:37.296924 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } attempt: 1
I20260619 20:31:37.297348 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.297213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:58994)
0619 20:31:37.297273 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.297317 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 20:31:37.297740 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.297657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:59000)
0619 20:31:37.297697 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.297722 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 20:31:37.297851 12357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:38975: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
I20260619 20:31:37.298087 12439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.151.20:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.151.62:36493,127.21.151.61:38335,127.21.151.60:38975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/krb5kdc/kudu_127.21.151.60.keytab
--principal=kudu/127.21.151.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.151.60:38975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.151.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 19:22:44 UTC on ccycloud.quasar-pzqoph.root.comops.site
build id 4456546
I20260619 20:31:37.298354 12439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 20:31:37.298607 12439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 20:31:37.303411 12445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 20:31:37.303447 12447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:37.303436 12439 server_base.cc:1034] running on GCE node
W20260619 20:31:37.303541 12444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 20:31:37.303853 12439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 20:31:37.304165 12439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 20:31:37.305334 12439 hybrid_clock.cc:648] HybridClock initialized: now 1781901097305333 us; error 40 us; skew 500 ppm
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 20:31:37.307893 12439 init.cc:377] Logged in from keytab as kudu/127.21.151.60@KRBTEST.COM (short username kudu)
I20260619 20:31:37.308287 12439 webserver.cc:460] Webserver started at http://127.21.151.60:38231/ using document root <none> and password file <none>
I20260619 20:31:37.308480 12439 fs_manager.cc:359] Metadata directory not provided
I20260619 20:31:37.308521 12439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 20:31:37.308609 12439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 20:31:37.309636 12439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/data/instance:
uuid: "c850cdb14f5f4ecf953ae70470ed8719"
format_stamp: "Formatted at 2026-06-19 20:31:37 on dist-test-slave-f9rk"
server_key: "27fc73075ed4186683892468348bef63"
server_key_iv: "162750b8c2bdf9aa496b7799334dc7cb"
server_key_version: "encryptionkey@0"
I20260619 20:31:37.309960 12439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/wal/instance:
uuid: "c850cdb14f5f4ecf953ae70470ed8719"
format_stamp: "Formatted at 2026-06-19 20:31:37 on dist-test-slave-f9rk"
server_key: "27fc73075ed4186683892468348bef63"
server_key_iv: "162750b8c2bdf9aa496b7799334dc7cb"
server_key_version: "encryptionkey@0"
I20260619 20:31:37.311267 12439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 20:31:37.312088 12454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 20:31:37.312319 12439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 20:31:37.312405 12439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/data,/tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/wal
uuid: "c850cdb14f5f4ecf953ae70470ed8719"
format_stamp: "Formatted at 2026-06-19 20:31:37 on dist-test-slave-f9rk"
server_key: "27fc73075ed4186683892468348bef63"
server_key_iv: "162750b8c2bdf9aa496b7799334dc7cb"
server_key_version: "encryptionkey@0"
I20260619 20:31:37.312464 12439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 20:31:37.330339 12432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } attempt: 2
I20260619 20:31:37.330883 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.330756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:59002)
0619 20:31:37.330818 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.330855 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 20:31:37.331220 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.331141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:59014)
0619 20:31:37.331182 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.331205 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 20:31:37.331326 12432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.151.60:38975: Network error: Client connection negotiation failed: client connection to 127.21.151.60:38975: connect: Connection refused (error 111)
I20260619 20:31:37.331687 12439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 20:31:37.333645 12439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 20:31:37.333820 12439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 20:31:37.337680 12439 rpc_server.cc:307] RPC server started. Bound to: 127.21.151.60:38975
I20260619 20:31:37.337714 12506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.151.60:38975 every 8 connection(s)
I20260619 20:31:37.338078 12439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/data/info.pb
I20260619 20:31:37.339057 12507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 20:31:37.340669 12507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:37.346750 22108 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM6qw9r/build/release/bin/kudu as pid 12439
I20260619 20:31:37.346830 22108 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM6qw9r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781900829246020-22108-0/minicluster-data/master-2/wal/instance
I20260619 20:31:37.347054 22108 external_mini_cluster.cc:1428] Setting key 0dd6592d74fe324ca9a30e421ea1c549
I20260619 20:31:37.349673 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.341360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.21.151.62:36493)
0619 20:31:37.341428 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.341430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.341653 (+ 223us) server_negotiation.cc:407] Connection header received
0619 20:31:37.342153 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.342154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.342166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.342230 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.342847 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.343316 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.344365 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.344556 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.344643 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.346648 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.346650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.346650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.346656 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.348302 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.348775 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.348776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.348777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.348837 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.349070 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.349071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.349072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.349202 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.349268 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.349496 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.349584 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 20:31:37.349820 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.341349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.62:36493 (local address 127.0.0.1:39988)
0619 20:31:37.341531 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.341567 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:37.342041 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:37.342310 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:37.342313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:37.342548 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:37.342745 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.342751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.343421 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.343422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.344260 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.344263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.344685 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.344687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.344801 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.345243 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:37.345251 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:37.346544 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:37.348385 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.348392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.348394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.348675 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.348912 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.348913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.348913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.349009 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.349264 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:37.349268 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:37.349376 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 20:31:37.349458 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 20:31:37.349609 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260619 20:31:37.350387 12507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:31:37.356467 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.347400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.21.151.60:38975)
0619 20:31:37.347589 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.347594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.347768 (+ 174us) server_negotiation.cc:407] Connection header received
0619 20:31:37.347818 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.347821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.347836 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.347926 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.348549 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.349497 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.350512 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.350893 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.350982 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.353244 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.353251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.353252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.353259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.355106 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.355538 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.355542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.355542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.355604 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.355820 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.355821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.355821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.356024 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.356174 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.356273 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.356363 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260619 20:31:37.357247 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.350627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.61:38335 (local address 127.0.0.1:41614)
0619 20:31:37.350681 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.350691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:37.350735 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:37.350853 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:37.350853 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:37.350982 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:37.351022 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.351024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.351653 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.351654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.352129 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.352131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.352518 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.352519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.352585 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.352984 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:37.352989 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:37.354294 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:37.356143 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.356144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.356145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.356423 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.356634 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.356635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.356635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.356702 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.356958 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:37.356959 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:37.356998 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 20:31:37.357064 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 20:31:37.357151 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 20:31:37.357292 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.350693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.21.151.61:38335)
0619 20:31:37.350741 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.350743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.350768 (+ 25us) server_negotiation.cc:407] Connection header received
0619 20:31:37.350792 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.350793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.350801 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.350842 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.351100 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.351544 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.352212 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.352431 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.352517 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.354383 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.354385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.354385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.354392 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.356068 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.356506 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.356508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.356508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.356571 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.356758 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.356759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.356759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.356875 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.356955 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.357108 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.357196 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:31:37.357632 12507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.60@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.62@KRBTEST.COM
I20260619 20:31:37.364455 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.357919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.21.151.60:38975)
0619 20:31:37.357954 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.357956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.357964 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:31:37.358009 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.358010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.358020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.358077 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.358283 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.358919 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.359715 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.360015 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.360074 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.361797 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.361798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.361798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.361803 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.363232 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.363601 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.363602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.363602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.363697 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.363935 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.363936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.363936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.364034 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.364085 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.364258 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.364370 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":17}
I20260619 20:31:37.364467 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.357883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:59044)
0619 20:31:37.357927 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.357936 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:37.357971 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:37.358079 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:37.358079 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:37.358182 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:37.358210 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.358211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.359008 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.359009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.359576 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.359578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.360082 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.360083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.360137 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.360553 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:37.360557 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:37.361686 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:37.363311 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.363312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.363312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.363522 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.363800 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.363801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.363801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.363868 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.364104 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:37.364105 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:37.364155 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 20:31:37.364234 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 20:31:37.364328 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 20:31:37.365856 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.357296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.21.151.62:36493)
0619 20:31:37.357337 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.357340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.357828 (+ 488us) server_negotiation.cc:407] Connection header received
0619 20:31:37.357960 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.357961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.357974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.358025 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.358625 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.359220 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.360027 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.360332 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.360385 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.362680 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.362681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.362681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.362690 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.364459 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.365028 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.365029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.365030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.365108 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.365310 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.365311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.365311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.365465 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.365649 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.365687 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.365762 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 20:31:37.366413 12507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719: Bootstrap starting.
I20260619 20:31:37.367528 12507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:37.367859 12507 log.cc:824] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:37.368602 12507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719: No bootstrap required, opened a new log
I20260619 20:31:37.370028 12507 raft_consensus.cc:348] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } }
I20260619 20:31:37.370191 12507 raft_consensus.cc:729] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c850cdb14f5f4ecf953ae70470ed8719, State: Initialized, Role: FOLLOWER
I20260619 20:31:37.370278 12507 consensus_queue.cc:260] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } }
I20260619 20:31:37.370677 12513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } } }
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.151.61@KRBTEST.COM
I20260619 20:31:37.370731 12507 sys_catalog.cc:564] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:37.370788 12513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 20:31:37.372669 12524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:31:37.372745 12524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:31:37.372982 12357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } attempt: 2
I20260619 20:31:37.374321 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.366555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.21.151.61:38335)
0619 20:31:37.366605 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.366607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.366615 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:31:37.366637 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.366638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.366647 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.366706 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.367381 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.367954 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.368630 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.368840 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.368915 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.371349 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.371350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.371351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.371358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.373144 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.373520 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.373521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.373521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.373580 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.373800 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.373801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.373802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.373934 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.374084 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.374154 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.374233 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 20:31:37.375844 22108 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 20:31:37.373116 12439 master_runner.cc:186] Error getting master registration for 127.21.151.62:36493: OK, instance_id { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" instance_seqno: 1781901097030802 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:31:37.376333 12439 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:31:37.377048 12331 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c8f2fa3b0be844ef8efaab96579f0b62"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40004
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.62@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
I20260619 20:31:37.382735 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.375862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.21.151.60:38975)
0619 20:31:37.375903 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.375905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.375913 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:31:37.375981 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.375982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.375991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.376055 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.376390 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.376939 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.377892 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.378184 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.378260 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.380193 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.380194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.380194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.380201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.381569 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.381937 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.381939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.381939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.381996 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.382196 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.382197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.382197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.382332 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.382413 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.382578 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.382649 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 20:31:37.382738 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.375779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:59048)
0619 20:31:37.375866 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.375877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:37.375921 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:37.376099 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:37.376100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:37.376250 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:37.376307 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.376309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.377018 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.377019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.377789 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.377791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.378268 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.378269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.378331 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.378738 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:37.378743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:37.380126 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:37.381676 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.381677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.381678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.381851 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.382071 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.382072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.382072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.382134 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.382410 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:37.382411 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:37.382457 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 20:31:37.382538 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 20:31:37.382638 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 20:31:37.384112 12357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62: Bootstrap starting.
I20260619 20:31:37.384678 12357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:37.384913 12357 log.cc:824] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:37.385622 12357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62: No bootstrap required, opened a new log
I20260619 20:31:37.386575 12357 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } }
I20260619 20:31:37.386685 12357 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f2fa3b0be844ef8efaab96579f0b62, State: Initialized, Role: FOLLOWER
I20260619 20:31:37.386744 12357 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } }
I20260619 20:31:37.387007 12357 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:37.387101 12331 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 20:31:37.387193 12331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:37.388008 12331 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } }
I20260619 20:31:37.388561 12481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c850cdb14f5f4ecf953ae70470ed8719"
I20260619 20:31:37.388687 12481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [term 0 FOLLOWER]: Advancing to term 1
W20260619 20:31:37.388805 12538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:31:37.388851 12538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:31:37.389616 12481 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f2fa3b0be844ef8efaab96579f0b62 in term 1.
I20260619 20:31:37.389844 12306 leader_election.cc:302] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c850cdb14f5f4ecf953ae70470ed8719, c8f2fa3b0be844ef8efaab96579f0b62; no voters:
I20260619 20:31:37.387023 12527 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } } }
I20260619 20:31:37.390058 12527 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:37.390163 12527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 20:31:37.390228 12527 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [term 1 LEADER]: Becoming Leader. State: Replica: c8f2fa3b0be844ef8efaab96579f0b62, State: Running, Role: LEADER
I20260619 20:31:37.390300 12527 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } }
I20260619 20:31:37.390739 12527 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f2fa3b0be844ef8efaab96579f0b62. Latest consensus state: current_term: 1 leader_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } } }
I20260619 20:31:37.390854 12527 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:37.391036 12542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 20:31:37.391215 12542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 20:31:37.388255 12331 leader_election.cc:288] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [CANDIDATE]: Term 1 election: Requested vote from peers 3a33503a99564da18cf567d1f5a59a07 (127.21.151.61:38335), c850cdb14f5f4ecf953ae70470ed8719 (127.21.151.60:38975)
I20260619 20:31:37.389456 12289 master_runner.cc:186] Error getting master registration for 127.21.151.61:38335: OK, instance_id { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" instance_seqno: 1781901097185093 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 20:31:37.393139 12289 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 20:31:37.394428 12481 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [term 1 FOLLOWER]: Refusing update from remote peer c8f2fa3b0be844ef8efaab96579f0b62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:37.394783 12527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:37.396171 12513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f2fa3b0be844ef8efaab96579f0b62. Latest consensus state: current_term: 1 leader_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } } }
I20260619 20:31:37.396284 12513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:37.409179 12548 mvcc.cc:205] Tried to move back new op lower bound from 7298666894925430784 to 7298666894911410176. Current Snapshot: MvccSnapshot[applied={T|T < 7298666894925430784}]
I20260619 20:31:37.409770 12527 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } } }
I20260619 20:31:37.409878 12527 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [sys.catalog]: This master's current role is: LEADER
I20260619 20:31:37.410213 12513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } } }
I20260619 20:31:37.410302 12513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:37.410501 12551 mvcc.cc:205] Tried to move back new op lower bound from 7298666894925430784 to 7298666894911410176. Current Snapshot: MvccSnapshot[applied={T|T < 7298666894925430784}]
I20260619 20:31:37.411101 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.393209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.21.151.61:38335)
0619 20:31:37.393261 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.393265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.399853 (+ 6588us) server_negotiation.cc:407] Connection header received
0619 20:31:37.399919 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.399921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.399934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.399966 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.401001 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.401726 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.404362 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.404674 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.404741 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.406906 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.406908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.406908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.406917 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.408551 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.409654 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.409655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.409655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.409726 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.410205 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.410207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.410207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.410353 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.410422 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.410922 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.411004 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:31:37.413378 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.393452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.21.151.62:36493)
0619 20:31:37.393494 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.393496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.400127 (+ 6631us) server_negotiation.cc:407] Connection header received
0619 20:31:37.400228 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.400229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.400241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.400306 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.401619 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.402353 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.403409 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.403800 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.403879 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.405963 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.405964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.405965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.405971 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.407586 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.409356 (+ 1770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.409357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.409358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.409415 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.412653 (+ 3238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.412654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.412654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.412806 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.412891 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.413204 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.413287 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:31:37.414079 12542 catalog_manager.cc:1347] Generated new cluster ID: 4f893fef6a0d400095e628b268c6ee32
I20260619 20:31:37.414132 12542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 20:31:37.419543 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.393524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.21.151.60:38975)
0619 20:31:37.393558 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.393559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.404709 (+ 11150us) server_negotiation.cc:407] Connection header received
0619 20:31:37.406305 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.406306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.406317 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.406346 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.410770 (+ 4424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.411490 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.412491 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.412810 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.412876 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.416079 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.416081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.416081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.416089 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.417717 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.418403 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.418404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.418405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.418465 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.418780 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.418781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.418782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.418938 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.419034 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.419360 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.419445 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 20:31:37.422452 12432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } attempt: 3
Jun 19 20:31:37 dist-test-slave-f9rk krb5kdc[12276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781901097, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.151.61@KRBTEST.COM for kudu/127.21.151.60@KRBTEST.COM
I20260619 20:31:37.429984 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.422773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.151.60:38975 (local address 127.0.0.1:59068)
0619 20:31:37.422859 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 20:31:37.422869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 20:31:37.422904 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 20:31:37.423107 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 20:31:37.423108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 20:31:37.423278 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 20:31:37.423322 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.423324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.424225 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.424225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.425008 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 20:31:37.425009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.425462 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.425463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 20:31:37.425513 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.425961 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 20:31:37.425966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 20:31:37.427369 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 20:31:37.429042 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.429043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.429043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.429238 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.429418 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 20:31:37.429418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 20:31:37.429419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 20:31:37.429465 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 20:31:37.429692 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 20:31:37.429693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 20:31:37.429734 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 20:31:37.429792 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 20:31:37.429884 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 20:31:37.430321 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:37.422964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.21.151.60:38975)
0619 20:31:37.423002 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.423003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.423009 (+ 6us) server_negotiation.cc:407] Connection header received
0619 20:31:37.423026 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.423027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.423034 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.423072 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.423388 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.424140 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.425081 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:37.425366 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:37.425416 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:37.427453 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 20:31:37.427454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 20:31:37.427455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 20:31:37.427461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 20:31:37.428958 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.429316 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.429316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.429317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.429363 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 20:31:37.429516 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 20:31:37.429517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 20:31:37.429517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 20:31:37.429626 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 20:31:37.429672 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:37.430158 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 20:31:37.430227 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:31:37.432251 12432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07: Bootstrap starting.
I20260619 20:31:37.433007 12432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07: Neither blocks nor log segments found. Creating new log.
I20260619 20:31:37.433305 12432 log.cc:824] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07: Log is configured to *not* fsync() on all Append() calls
I20260619 20:31:37.435962 12432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07: No bootstrap required, opened a new log
I20260619 20:31:37.438037 12432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } }
I20260619 20:31:37.438179 12432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a33503a99564da18cf567d1f5a59a07, State: Initialized, Role: FOLLOWER
I20260619 20:31:37.438258 12432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } }
I20260619 20:31:37.438601 12432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 20:31:37.439707 12405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 20:31:37.440650 12405 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [term 1 FOLLOWER]: Refusing update from remote peer c8f2fa3b0be844ef8efaab96579f0b62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 20:31:37.440836 12406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3a33503a99564da18cf567d1f5a59a07"
I20260619 20:31:37.441573 12406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f2fa3b0be844ef8efaab96579f0b62 in term 1.
I20260619 20:31:37.442117 12554 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 20:31:37.443004 12542 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 20:31:37.443782 12542 catalog_manager.cc:1504] Loading token signing keys...
I20260619 20:31:37.445724 12540 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } } }
I20260619 20:31:37.445824 12540 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62 [sys.catalog]: This master's current role is: LEADER
W20260619 20:31:37.446875 12572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 20:31:37.446974 12572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 20:31:37.447120 12568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } } }
I20260619 20:31:37.447258 12568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:37.447366 12568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f2fa3b0be844ef8efaab96579f0b62. Latest consensus state: current_term: 1 leader_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } } }
I20260619 20:31:37.447575 12568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:37.447680 12568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f2fa3b0be844ef8efaab96579f0b62" member_type: VOTER last_known_addr { host: "127.21.151.62" port: 36493 } } peers { permanent_uuid: "3a33503a99564da18cf567d1f5a59a07" member_type: VOTER last_known_addr { host: "127.21.151.61" port: 38335 } } peers { permanent_uuid: "c850cdb14f5f4ecf953ae70470ed8719" member_type: VOTER last_known_addr { host: "127.21.151.60" port: 38975 } } }
I20260619 20:31:37.447798 12568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 20:31:37.448532 12566 mvcc.cc:205] Tried to move back new op lower bound from 7298666895116591104 to 7298666894911410176. Current Snapshot: MvccSnapshot[applied={T|T < 7298666894925430784}]
I20260619 20:31:37.451328 12542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8f2fa3b0be844ef8efaab96579f0b62: Generated new TSK 0
I20260619 20:31:37.451493 12542 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 20:31:37.469789 12360 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.468458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.21.151.62:36493)
0619 20:31:37.468502 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.468504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.469081 (+ 577us) server_negotiation.cc:407] Connection header received
0619 20:31:37.469198 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.469199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.469211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.469242 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.469704 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40018: BlockingRecv error: recv got EOF from 127.0.0.1:40018 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 20:31:37.470513 12511 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.468839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.21.151.60:38975)
0619 20:31:37.468884 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.468886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.470025 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 20:31:37.470122 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.470123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.470132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.470161 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.470454 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59082: BlockingRecv error: recv got EOF from 127.0.0.1:59082 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 20:31:37.471448 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.470836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.21.151.61:38335)
0619 20:31:37.470880 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.470882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.470964 (+ 82us) server_negotiation.cc:407] Connection header received
0619 20:31:37.471056 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.471057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.471066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.471096 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.471392 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41652: BlockingRecv error: recv got EOF from 127.0.0.1:41652 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:31:37.476702 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.475985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.21.151.61:38335)
0619 20:31:37.476023 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.476024 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.476194 (+ 170us) server_negotiation.cc:407] Connection header received
0619 20:31:37.476295 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.476296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.476314 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.476342 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.476652 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41664: BlockingRecv error: recv got EOF from 127.0.0.1:41664 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 20:31:37.479846 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.479043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.21.151.61:38335)
0619 20:31:37.479076 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.479078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.479281 (+ 203us) server_negotiation.cc:407] Connection header received
0619 20:31:37.479385 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.479386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.479395 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.479422 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.479795 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41674: BlockingRecv error: recv got EOF from 127.0.0.1:41674 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 20:31:37.483857 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.483006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.21.151.61:38335)
0619 20:31:37.483052 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.483054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.483278 (+ 224us) server_negotiation.cc:407] Connection header received
0619 20:31:37.483380 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.483381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.483391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.483419 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.483813 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41684: BlockingRecv error: recv got EOF from 127.0.0.1:41684 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 20:31:37.489770 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.489020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.21.151.61:38335)
0619 20:31:37.489054 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.489056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.489267 (+ 211us) server_negotiation.cc:407] Connection header received
0619 20:31:37.489359 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.489360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.489370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.489399 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.489710 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41688: BlockingRecv error: recv got EOF from 127.0.0.1:41688 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 20:31:37.500031 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.499086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.21.151.61:38335)
0619 20:31:37.499131 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.499133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.499381 (+ 248us) server_negotiation.cc:407] Connection header received
0619 20:31:37.499488 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.499489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.499501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.499531 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.499970 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41692: BlockingRecv error: recv got EOF from 127.0.0.1:41692 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:31:37.518373 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.517391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.21.151.61:38335)
0619 20:31:37.517435 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.517437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.517776 (+ 339us) server_negotiation.cc:407] Connection header received
0619 20:31:37.517884 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.517886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.517898 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.517928 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.518311 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41704: BlockingRecv error: recv got EOF from 127.0.0.1:41704 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:31:37.552723 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.551830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.21.151.61:38335)
0619 20:31:37.551874 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.551877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.552110 (+ 233us) server_negotiation.cc:407] Connection header received
0619 20:31:37.552220 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.552221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.552231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.552260 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.552654 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41714: BlockingRecv error: recv got EOF from 127.0.0.1:41714 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 20:31:37.618917 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.618221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.21.151.61:38335)
0619 20:31:37.618273 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.618276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.618405 (+ 129us) server_negotiation.cc:407] Connection header received
0619 20:31:37.618439 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.618440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.618452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.618512 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.618856 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41716: BlockingRecv error: recv got EOF from 127.0.0.1:41716 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 20:31:37.749039 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:37.748225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.21.151.61:38335)
0619 20:31:37.748325 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:37.748327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:37.748421 (+ 94us) server_negotiation.cc:407] Connection header received
0619 20:31:37.748514 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:37.748515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:37.748526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:37.748602 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:37.748974 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41730: BlockingRecv error: recv got EOF from 127.0.0.1:41730 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 20:31:38.006958 12437 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 20:31:38.006217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.21.151.61:38335)
0619 20:31:38.006291 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:38.006293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:38.006371 (+ 78us) server_negotiation.cc:407] Connection header received
0619 20:31:38.006434 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:38.006435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:38.006445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:38.006507 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 20:31:38.006877 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41744: BlockingRecv error: recv got EOF from 127.0.0.1:41744 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 20:31:38.373178 12524 catalog_manager.cc:1259] Loaded cluster ID: 4f893fef6a0d400095e628b268c6ee32
I20260619 20:31:38.373274 12524 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719: loading cluster ID for follower catalog manager: success
I20260619 20:31:38.374071 12524 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719: acquiring CA information for follower catalog manager: success
I20260619 20:31:38.374380 12524 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c850cdb14f5f4ecf953ae70470ed8719: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:31:38.447556 12572 catalog_manager.cc:1259] Loaded cluster ID: 4f893fef6a0d400095e628b268c6ee32
I20260619 20:31:38.447672 12572 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07: loading cluster ID for follower catalog manager: success
I20260619 20:31:38.448590 12572 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07: acquiring CA information for follower catalog manager: success
I20260619 20:31:38.448912 12572 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a33503a99564da18cf567d1f5a59a07: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 20:31:38.522949 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:38.520364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.21.151.61:38335)
0619 20:31:38.520485 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:38.520488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:38.520559 (+ 71us) server_negotiation.cc:407] Connection header received
0619 20:31:38.520618 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:38.520619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:38.520671 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:38.520737 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:38.520923 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:38.521477 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:38.522242 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:38.522436 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:38.522487 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:38.522623 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:38.522730 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:38.522788 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:38.522860 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 20:31:38.522876 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260619 20:31:38.526870 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:38.524742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.21.151.60:38975)
0619 20:31:38.524866 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:38.524868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:38.524875 (+ 7us) server_negotiation.cc:407] Connection header received
0619 20:31:38.524910 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:38.524910 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:38.524944 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:38.525013 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:38.525097 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:38.525556 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:38.526230 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:38.526419 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:38.526469 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:38.526597 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:38.526697 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:38.526759 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:38.526789 (+ 30us) server_negotiation.cc:299] Negotiation successful
0619 20:31:38.526806 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260619 20:31:38.531284 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:38.528693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.21.151.61:38335)
0619 20:31:38.528738 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:38.528740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:38.528748 (+ 8us) server_negotiation.cc:407] Connection header received
0619 20:31:38.528773 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:38.528774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:38.528783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:38.528864 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:38.528991 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:38.529731 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:38.530581 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:38.530792 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:38.530878 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:38.531029 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:38.531063 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:38.531088 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:38.531179 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 20:31:38.531208 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 20:31:38.531662 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:38.528702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.21.151.60:38975)
0619 20:31:38.528742 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:38.528744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:38.528947 (+ 203us) server_negotiation.cc:407] Connection header received
0619 20:31:38.529009 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:38.529010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:38.529019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:38.529074 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:38.529207 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:38.529648 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:38.530728 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:38.531017 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:38.531103 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:38.531248 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:38.531286 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:38.531329 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:38.531529 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 20:31:38.531550 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 20:31:38.532048 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 20:31:38.528588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.21.151.62:36493)
0619 20:31:38.528704 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 20:31:38.528903 (+ 199us) server_negotiation.cc:399] Waiting for connection header
0619 20:31:38.528912 (+ 9us) server_negotiation.cc:407] Connection header received
0619 20:31:38.528959 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 20:31:38.528960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 20:31:38.529010 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 20:31:38.529226 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 20:31:38.529264 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:38.529995 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:38.530954 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 20:31:38.531272 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 20:31:38.531364 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 20:31:38.531645 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 20:31:38.531760 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 20:31:38.531825 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 20:31:38.531890 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 20:31:38.531925 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260619 20:31:38.533490 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 12289
I20260619 20:31:38.538820 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 12364
I20260619 20:31:38.543792 22108 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM6qw9r/build/release/bin/kudu with pid 12439
2026-06-19T20:31:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2768 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5560 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269304 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST